Hacker News Reader

Comment by moe | Hacker News Reader

naturally oblivious to the security implications

Rest assured Lenovo was perfectly aware of the security and privacy implications of this feature from the beginning.

They merely try to sound oblivious because their laywers hope that will soften the legal and media repercussions.

Honestly, I think that's unlikely. This is far too sloppy to have been intentional. There are much better ways to implement a backdoor when you control the OS image. This is just incompetence, plain and simple.

Superfish looks like the kind of crapware that pays OEMs to include it in their bundle. Lenovo took the cash and didn't bother to review the code. Superfish, for its part, probably doesn't have the best and brightest engineers working for them. They probably tasked a junior programmer with working around SSL, who then committed the first solution that worked without ever thinking about security implications, and they shipped it.

Cannot see how this could possibly be true. Having been privy to OS bundling for products, I can assure you there is lengthy contracts, and negotiations, about exactly what is happening. You do no simply walk up to Lenovo and have your "software" installed into the OS without a very detailed contract and pay structure. There also looks to be js injected into pages, which is serving up the ads, and a comment about Lenovo [1]. Think about what the means. There was a project at this company, where they had meetings, project plans, testing to make sure it worked, and a very detailed idea of what was going on. Never mind all the ramping up capacity due to new Lenovo's boxes coming on-line. There is zero chance this was some low level junior programmer fly by night operation.

[1] https://news.ycombinator.com/item?id=9072542

Oh I'm sure they had lots of meetings about the contracts and pay structure, and they may have done testing to make sure it didn't break things, but apparently no one did a security review. Sadly, this doesn't surprise me that much.

If they did know about the problem, they could have fixed it. If the app simply generated a new key as part of first-time use, then it would just be run-of-the-mill crapware rather than a gaping security hole. Even if Lenovo has malicious intent, it would still have been in their best interests to do at least that, yet they didn't. Hence I assume it was incompetence.

but apparently no one did a security review

It doesn't take a "security review" to spot a gaping security and privacy violation like this.

Any engineer with even the slightest clue of how a browser and "the internet" works would have called this out during the first "How does this product work?"-presentation.

Let's not pretend Lenovo is staffed with monkeys.

“Never ascribe to malice that which can adequately be explained by incompetence.”

Remember stuff like this:

http://www.cryptofails.com/post/70059600123/saltstack-rsa-e-...

(Which, possibly unfairly, is one reason I'm leaning more towards ansible than saltstack to this day -- I mean, if stuff like that got through... what else, in more complex areas of the system?)

The problem in Lenovo's situation is, calling it incompetence is the real stretch. You could call Charles Manson incompetent saying he just didn't know what he was doing was wrong, but everyone knows he was just evil.

Never falsely attribute to incompetence what is actually ascribable to malice. You can't come in here with a straight face and say that no one at Lenovo considered the security risk of including this software. If it was considered and they pushed ahead with it anyway, that's malice.

I don't think anyone there thought/realized that they were including a backdoor usable by any number of third parties (by virtue of installing a mitm-cert, and giving away the key). And this case is much worse than any other crapware-by-way-of-oem than I've heard of. But given the amount of nasty stuff most vendors seem to install on systems -- it appears to me that no one really looks at what is installed, or gives much thought to the consequences.

It's negligent, and in this case probably criminally so -- and that might constitute "an evil" -- but I don't think this is the result of someone's overt intentional evil act. I don't think anyone actually did consider the security risk of this particular piece of software. Maybe I'm naive, but if nothing else, the risk of lawsuits/backlash seems too great in this case.

I don't like ads and bloatware, but I think calling them "evil" is diluting what "evil" means.

I might be wrong, of course. But I don't think any of the big OEMs does any real review of the crap that is installed on computers -- and I think forgetting to generate an unique cert/key on post-install/first run is an error -- not intentional. Deciding to install this kind of crap strikes me as a very poor decision -- but I'm still not sure I'd consider it evil. Evil would be using the Intel management co-prosessor to do something similar -- presumably then a clean install wouldn't help.

But that argument means either that these companies do not have a security team (we know they do), that the security team signed off on this (we know they wouldn't), or the security team raised the risk and management chose to ignore it. There's absolutely no option that says "no one ever thought of this risk", at least not in the world we live in. I've worked in enterprise security and I still work in the security industry. There is just no way that this software got approved to be put in a default install and had no review from the security department.

That's what I meant by invoking the opposite of Hanlon's razor. Sure, never attribute to malice what can be explained by ignorance. But my point is, you can't explain this one with ignorance. There is just no way that Lenovo has hired a security team that would do a review of this and say it looks fine, and no way a company the size and stature of Lenovo would not have a competent security team. The only logical answer is that this was raised as a risk and management chose to accept the risk.

I'm not saying they're evil (I used that word to describe Charles Manson), nor that their end goal was for users to be compromised. Merely that they had to know this was a bad idea, and they chose to do it anyway.

You may be right. I'm inclined to believe the provisioning team in Lenovo is understaffed, and that they don't really do much security analysis at all. So I believe their negligent, and that their process is negligent. But I'm open to the idea that I might very well be wrong about that. Either way, it doesn't speak very highly of what kind of quality one can expect to get when shopping Lenovo products.

I generally agree, but this is a situation that can be explained by either an embarrassing level of incompetence or a pretty minor amount of malice (or even indifference). So I'll assume malice until I see them own up to that much incompetence.

Never exclusively ascribe either malice or incompetence to explain the actions of a large bureaucracy. It is nearly always both.

You're so optimistic it hurts

"Any engineer" means something in HN, but we're not talking about "people who read HN" levels of engineer here, don't be mistaken.

Some people that have had no or limited experience with software are assigned to software projects, and that's the issue with companies like Lenovo.

Operations the size of Lenovo have a fairly intense vetting process before a product goes to market.

I find it very hard to believe that no red flags were raised by any of the engineers, managers and especially lawyers who must have screened this "feature" for problems.

It seems more plausible that the problem was known from the beginning (it is by design after all) and Lenovo decided to risk it.

My own experience makes me suspect the same thing. I used to work for a company that was, at the time, trying to develop a privacy-enhancing product (ironically enough...) which did something somewhat similar (although not on the size of this fuckup -- they'd be intercepting, but not tampering with, encrypted traffic, and storing encrypted private data).

Virtually everyone in the engineering team raised a flag when the imbec...uhm, the Product Manager came up with the idea. We pointed out that a) this burdens us with the responsibility of storing sensitive data which can, at least, have significant legal implications and that b) even if it's encrypted data, it may be a little hard to market a privacy device that works by uploading user data to our server as a first step without being transparent about the whole process. Oh, and c) that the data recovery mechanism he proposed (which involved storing the users' private keys on our servers as well, just in case they lost their precious little gimmick) was, in this case, entirely retarded.

The whole thing didn't even make it to Legal, because everyone in the decision tree just thought that since there's no plaintext data being stored, there's no potential for a lawsuit (and when we told the PM about Lavabit, he came back two hours later saying he Googled it and that we're covered since we're not an e-mail provider). The bright heads in Marketing weren't exactly sure about the whole transparency thing. They thought we should keep it simple and just tell people that their data is safely encrypted and be done with it, because end-users don't need to know about tech mumbo-jumbo like encryption keys and all that.

I don't work there anymore (thank God) and they haven't launched in the meantime, but when I left, they were basically working on implementing this clusterfuck.

I'm sorry I can't be more specific than this (for obvious reasons, I hope). The point is, however, that decisions as complex as these (there's a stack of paperwork thicker than the Osbourne-1 involved in preloading anything on a laptop) are made through an elaborate process, not made "by mistake".

Someone knew there was a problem. The problem may have ended up misunderstood or washed out along the decision chain (although I find that fairly unlikely), but someone, at some point, decided this was ok.

Once one vendor in your space says "we filter HTTPS traffic for nasty viruses!", it becomes a marketing weapon, and lots of customers think "well, why should I go with A when B protects me better?"

> Operations the size of Lenovo have a fairly intense vetting process before a product goes to market.

How does that go along with a gigantic fuckup like this? Ipso facto there was no vetting, otherwise this wouldn't happen. What did they expect, that this wouldn't come out, that this wouldn't damage their brand even further? If it was done out of malice it is still poorly vetted and incompetent malice.

Just repeat, “Never ascribe to malice that which can adequately be explained by incompetence.”

They probably didn't figure out that anyone would have a problem with this. For them, it's just a cool gimmick to get some money. That it is a gaping security hole which makes about 0.42 % of user population mad, probably never occurred to them.

Unfortunately, for the 0.42 % (that is us, reading this site, and people of similar interests) it will be hard going to explain to the next 4.2 % why this is so bad. The remaining approximately 96 % of population will stay largely uninterested.

> Just repeat

Yea, read again. I claim that even if there was malice there necessarily was an element of incompetence present in that case as well.

> it will be hard going to explain to the next 4.2 % why this is so bad

Why? People aren't interested in exact details, that's why they rely on 0.42%. You can illustrate the magnitudes of moronity required to design some of their products and lack of respect for security by explaining that they approach those that are needed to drive a car which has chainsaw strapped on its steering wheel. This isn't mere buffer-overflows due to bad coding, these are comatose levels of stupidity.

Hopefully we .42 will inform our fellow 4.2ers when they come to us for advice when buying a new laptop/anything Lenovo makes. I don't think it will be so hard to explain it to them. They already know what adware is. Just mention it comes installed ready to track you. Always listening while you're visiting bank.com.

I doubt the usual lawyer assigned to this understands SSL and certificates well enough to say anything about it. They worry mostly about contracts, and this is a technical thing.

How many engineers do you think were in the "how it works" meeting?

I don't know, I've worked on some large government projects where things like this could have possibly slipped through because an engineer or two thought it was a clever way to workaround the issue. Granted they should have known and may have known but I'm not convinced they had to have known.

They probably tasked a junior programmer with working around SSL

I don't think I've seen a junior anything who was informed and insightful enough to write a network proxy, including SSL support, and the necessary certificate work.

How could you add mitm functionality by mistake?

Because you call it "enhanced functionality featuring cloud services", not a "man in the middle attack".

And calling it enhanced is not always an unreasonable interpretation. For instance, take the case of a cheap mobile phone with a very limited bandwidth. You can increase the end user satisfaction considerably if you move some of the functionality to a server layer so that when you browse, the things actually happen somewhere in a cloud and your phone is just displaying the result, without being the actual browser as seen by the site you visit.

Nokia did this with some of the cheaper devices, and I think it was quite OK. It comes down to how much you trust that party, of course, and how critical your communication is.

I think you give them too much credit. This was probably a decision made by a non-technical group without input from a technical group (e.g. Marketing goes and does something without even thinking of contacting Engineering), and whoever slipstreamed it into the factory image just followed instructions unquestioningly. This will likely result in an eventual retraction and apology, and internal process improvements being made to prevent such things from happening again. Such things will eventually happen again because large orgs are inefficient and individual employees are frustrated by inefficiency, so they'll work around the protocols. Rinse & repeat.

Someone has posted the actual script elsewhere in this thread [1]. Of particular interest is line 194:

  if (location.protocol === 'https:' && queryString.search(/dlsource=hdrykzc/i) !== -1) // Patch for Lenovo - do not run on https sites

So yes, it seems someone at Lenovo was security-aware enough to demand an exception for HTTPS. Unfortunately the fine folks at Superfish either didn't understand or didn't care.

[1] https://news.ycombinator.com/item?id=9072542

No, this is an example of the Lenovo sales / marketing people making distribution deals with dodgy third-party companies. The people who design the machines don't make the decision to ship MITM proxies on them.

I honestly don't know why Lenovo (and others) still make these third party deals. Just ship the machine with a blank OS, or install a vetted selection of open-source software (7zip, VLC, LibreOffice if they want). Just don't install crapware for the mediocre kickback it generates!

For low-end machines these bundling deals likely form a sizeable chunk of the profit margin. (I've heard eyebrow-raising numbers for e.g. the default browser spot.)

Yep. The other chunk results from the OEM's refusal to stick to any long term consistency in the components they spec in consumer lines of devices. In business lines, you will likely get a 6-12 month guarantee with a 6-24mo forecast showing exactly what is shipping with what (CPUs, GPUs, screens, hard drives, etc). With consumer lines, they change components & suppliers any time, for any reason.

>With consumer lines, they change components & suppliers any time, for any reason.

I always love when the same model (down to the part number) comes with a different configuration and board inside the case.

Reply on news.ycombinator.com