Hacker News Reader

> The biggest customer threw a gigantic hysterical fit and practically demanded to return the old root password which they knew. We just shrugged and complied. It's not like it will be ours network which can be breached. If they want to operate insecurely, they are welcome.

Did a PM or sales engineer from your org ever sit down with the customer to figure out what they were doing with root access that your solution lacked?

My naive read here is that the customer saw it as easier to keep root access because your product didn't expose what they wanted or did it in a cumbersome way.

They were providing root either way, the only change I assume was switching from the same root on all machines to a per-machine root password.

I can understand how that would be a pain in the ass if your legacy architecture (e.g. administration and reporting tooling, etc…) is predicated upon a unified root, you probably don’t even have a way to report the information so that machines and passwords could be paired.

Assuming a good way to provision keys is provided, I’d make the insecure unified root password into a paid option (with lots of warnings) the price of which would increase every year. This way the existing workflow of the customer keeps working short-term, but you CY the hell out of your A in case of breach at the customer’s, and at one point the line item will get large enough that the customer allocates time & money to fix their shit.

Charging for insecure systems incentivizes selling those same systems to customers.

Soon you’ll have sales guys regaling how this unified root password system is extremely convenient and you’re one of the few companies who provides this single-signature-security-system (S4 compliance). Next thing you know someone will gain an software patent on this to ensure no one else can provide it.

But charging for it also disincentivizes customers from asking for it.

Pass some law that says the money has to go to some federal cybersecurity fund or something to pay for cleaning up the mess.

> Soon you’ll have sales guys

The solution is trivial: no commission on this. Possibly even negative commission, can't convince your client not to take this? You get less money for the sale.

This way sales are incentivised to only talk about this if not doing so would lose them the sale entirely.

Yeah, they did want to maintain root access because they had started to rely on it, I'm not very familiar with exact details. We have an ongoing process of exposing different debug info in the restricted CLI, but it is a never ending task so they always want something which is only visible to the root.

Not OP, but over here, PM and sales usually understand that what's important to me and our other engineers (building a really great machine, with a well-architected, secure, extensible backend) is completely immaterial to my end customer.

They're trying to build a car or a chair or an appliance. To them, anything that gets in the way of that goal is bad, anything that supports it is good. Sure, the customer's engineering department had a grand vision of skilled shift leads empowered to look up the fault diagnostics, call over a QC engineer, and get two-party authorization to ignore the fault and continue...but once it hit the floor, and getting QC over to the machine took a whopping 4 minutes, the operators all memorized their lead's and QC's passwords and were rewarded for cutting TAKT times by 3 minutes and 50 seconds. Yes, bad product containment is a huge deal in the boardroom, but there's almost no will to enforce it on the production floor.

Just Get Shit Done and build a car that's controlled by a Russian botnet. Live laugh love that

That stuff for me is “entrepreneur porn”.

For b2b or b2c your customers don’t want to sit with you to make your product better.

Yes they want your product better.

Myself as a customer I don’t spend time filling in customer reviews. I just want to go on with my life.

I see the same in company I work for, our customers don’t have time - they have stuff to be done - either our software helps them achieve their goals or they leave.

You could use the excuse that you are complying with the law per "California passes law that bans default passwords in connected devices":

* https://techcrunch.com/2018/10/05/california-passes-law-that...

Is complying with the law an excuse these days?

Always has been.

Oh, I'm pretty sure we can just say whatever, even "we have decided and it will be so". There is only a tiny problem when a customer is bigger (and richer) than the vendor by an order of magnitude or more :) . They tend to control a lot, with such power imbalance.

Honestly doesn't get better in the reverse either. At Microsoft I'd sit in on calls with banks and other "big names", watching them pound the table and argue their way, usually involving something dangerous and borderline malicious use of the product, all the while thinking "we could buy your company with our cash on hand".

I think the key was that they're used to being the big dog in the room, and Microsoft made a point of not correcting them when that behavior was seen.

You are conflating two separate types of passwords.

Hardcoded password is different from default passwords. You can't change a hardcoded password, it's always part of the source code/image.

A default password can be changed.

Both can be backdoors if kept secret, but a hardcoded password from the onset is a backdoor.

Do you mean can't be changed on a live system? It's very easy to change a hard-coded password, just recompile the source and flash. That could definitely be what OP is referring to if they pushed OTA firmware upgrades.

There is nothing as permanent as a temporary fix...and worse people will become dependent on the specific inner workings of the temporary fix. Once given...

Relevant XKCD: "Every change breaks someone's workflow."

https://xkcd.com/1172/

Or instead of changing out of the blue what exists today and is probably soaked into whole following ecosystem around the product and would be massive PITA to change it as well, you could have create parallel way for login (i.e. via certificate) and tell customer that password access will be deprecated in 1-2 years, so they have time to update their tooling.

Can confirm the hardware vendor that I work for does the same. root password is given and customers are asked to change. No one wants to. Heck I would prefer if they disabled root login completely and used keys instead. But thats now it is. Some customers even enable login from web with those passwords. The stupidity is crazy.

Would be better to have such default creds parameterized by your build system. So every dev can choose theirs and you can choose unique ones per customer.

Meh, just prompt them to set/change it the first time they boot, and after that, it's their fault :)

Yeah, this is the only real solution.

Wait.

Do I understand this correctly? Are you saying you removed that root password, such that it wasn't possible to log in with another (but different) root password?

Or are you saying that you merely changed it, to one the customer no longer knew, but was the same password across all of your install base, for every customer?

How would that be any improvement, if the latter?

It was at that moment a simple change of the password. The benefit was that people outside of our company wouldn't have access (thus not sharing it further), and then as a future improvement we could have implemented some better key scheme. But while customers want to have root access it is not an option.

Theoretically we could implement a scheme with individual passwords or keys per device (as someone suggested in the comments above), but such thing is not being prioritized by our company, so that's our fault.

Jesus fuck.

This isn't a Latvian company, by chance, is it? Blink twice if yes.

The right scheme would be to require the user to set their own password on first login, and require that to happen before the device would work.

So you are intentionally including a secret backdoor in your software that is only known to some customers? Yikes!

What’s their complaint? Do they have a workflow that requires the root password all the time and it creates too much of a management to not have QWERTY1234 as root password?

Did you try to explain why this is a serious security issue?

Our lead developers tried, and they are way smarter than me). No luck, for some reason customers don't consider this a vulnerability. Even ones who take security more seriously and do some external pentesting on our equipment.

I've found having smart people explain things can be a detriment. Sometimes it needs to be the dumbest explanation to get people to understand.

Smart people explain things with big mental steps they deem obvious (true or not), and less smart people disagree with these steps but are unable to point the finger on them.

Something like handing out keys to your villa to every subcontractor that ever visits your house?

Unfortunately folding to an unreasonable (in terms of security) request like this emboldens them to be stupid in the future as well.

The customer is usually wrong.

> practically demanded to return the old root password

You could set that password just for that one customer, and there is no problem.

In this case, only one version out of three (the middle one) had this vuln, so it's very likely it was introduced by mistake. But your scenario happens all the time too.

I hope you guys got an indemnification agreement or something else that make you not liable when the inevitable happens but, knowing how these things work, I doubt it.