Backdoor in Zyxel Products | Hacker News Reader

Backdoor in Zyxel Products

eyecontrol.nl

184 pointsta98852 comments

> A hardcoded credential vulnerability was identified in the “zyfwp” user account in some Zyxel firewalls and AP controllers. The account was designed to deliver automatic firmware updates to connected access points through FTP.

Also translated as:

In the name of "improving security" we created a backdoor, which is known throughout the entire industry as "removing all security". We are very sorry we were caught so easily.

It does not take much to understand this was a terrible idea. Or that there are so many better ways to build and deploy automatic updates. You absolutely do not need to be able to control a machine remotely to deploy automatic updates. But counting the ways they went wrong building this update system makes it extremely hard to believe that was the intent.

> After a thorough investigation, we’ve identified the vulnerable products that are within their warranty and support period and are releasing firmware patches to address the issue, as shown in the table below.

That they went to the effort of identifying their minimal viable liability and all released patches for that, makes it even more disingenuous.

This is a company whose business is supposed to be about enhancing security. They should know better.

> 2020-12-02: Zyxel requests more information about how the vulnerability was discovered

Sorry, but how did you figure it out? Can we hide it better next time?

More like- did you violate any eula’s or laws in discovering this so we can sue you and call you a “convicted criminal hacker”

For something this severe, it should be made to everyone regardless of their current 'warranty and support period' status.

You'd be surprised, but it often comes to "nobody knows how to build that 10 year old piece of firmware anymore".

This. That's why I prefer to buy devices that have known compatibility with OpenWRT.

> It seemed the vulnerability had been introduced in the latest firmware version. Even though older versions do not have this vulnerability, they do have others (such as this buffer overflow) so you should still update.

>This is a company whose business is supposed to be about enhancing security. They should know better.

Except they don't.

Having owned a Zyxel NAS (comically named "NSA" back then), i can tell you their firmware, updates and support are even worse than you can imagine.

After this NAS, i vowed to never ever buy or touch any kind of Zyxel hardware again.

> You absolutely do not need to be able to control a machine remotely to deploy automatic updates.

Huh? How are you supposed to deploy the update then, or make it effective? I think the definition of "remotely control" differs somewhat from mine.

I could agree that you don't need to uniquely address a remote machine to deploy updates, but having control over the machine (if only for a moment) is the purpose of the update process.

"control remotely" as in "issue arbitrary commands" is absolutely _not_ required for an update mechanism.

The machine _itself_ can check for updates, and issue _only_ those commands that are _necessary_. You don't need complete control over a machine for an update mechanism.

But the entire update infrastructure has issues... Firmware binaries downloaded over unprotected basic FTP... No checksums... No authenticated signatures... And arbitrary commands issued at the root level.

> The machine _itself_ can check for updates, and issue _only_ those commands that are _necessary_.

The problem with this statement is that the machine doesn't know what sequence of actions is necessary to put the system into the "updated" state. If it knew, it wouldn't have to contact anything in order to update itself. It would already have known.

Where in „client pulls a signed image with a secure and trusted process from the update server, flashes it, then swaps it with its previous image“ does the update server command any control over the client?

That depends on the contents of the supplied image. But I agree, it's more about the need for the ability to take over control rather than exercising the ability.

> If it knew, it wouldn't have to contact anything in order to update itself. It would already have known.

Eh? That's how most package managers work. They only issue a very small number of commands to put them into an "updated" state. They don't generally do arbitrary things.

For example, if we take a look into aptitude, the deb packages it eventually fetches after running various authentications, contains:

+ A directory listing. (Contents-<arch>.gz)

+ Paths and checksums. (Release)

+ Signing information.

It doesn't have an arbitrary listing of commands to put it into the "updated" set - that's already decided by the format. The machine _does_ know the sequence of actions that is necessary. You don't need some kind of Turing complete monstrosity that can do absolutely anything to update it.

In the case of a _firmware update_ that's even simpler! Most of the time you're either replacing a kernel, or flashing the firmware to a pre-decided memory address. Both of those actions are decided long before you even begin issuing updates.

If you deploy a firmware update it contains new instructions that can do anything. The problem is the hardcoded credentials that an other 3rd party could dump from a device and then could be used to attack other devices. What they should have done if they wanted automatic updates was just have a public key on devices. They could then sign any updates with their private key.

Still with automatic updates you have to trust the vendor. However, with what they did updates/commands could be coming from anyone willing to dump device firmware.

Machines can pull rather than manufacturers push, using cryptographic signatures to verify origin. At no point during, say, a Linux embedded device doing an "apt-get upgrade" in a cron job, is anyone remotely controlling the machine.

I see, the difference is that I still consider it remote control even when the act of programming is separated by time from the act of execution.

...the all software in the world is a form of "remote control" on the initial installation.

I don't disagree with that :)

Let me summarize the issue:

"I was surprised to find a user account 'zyfwp' with a password hash"

"this account seemed to work on both the SSH and web interface"

"Globally, more than 100.000 devices have exposed their web interface to the internet"

Linux manages to auto update itself by fetching updates from the outside, when IT wants to do so.

Zyxel tried to add a function where THEY can access devices over the internet and do who-knows-what whenever they wanted.

And they did such a poor job of that, that they managed to give access to the devices to everybody.

It's a pity there are no long-term effects from the cases like those.

We've had a serious security vulnerability introduced in the firmware release, and there were no negative consequences for Zyxel at all, other than some bad press (and it's probably a fairly small amount of the bad press). What I would like to see is a post-mortem with mitigations -- how did this happen, and how will they prevent similar occurrences in the future. Instead, most likely nothing will happen. And I will not be surprised if the "firmware update account" will be back in the next release, probably still insecure, but hidden better so that random researchers won't stumble on it.

I wish there were some sort of liability for the manufacturers for the cases like those. For example, if a person runs a red light by accident and get caught, they get a fine and an insurance premium increase. We need something like this for IOT suppliers as well.

> but hidden better so that random researchers won't stumble on it.

> 2020-12-02: Zyxel requests more information about how the vulnerability was discovered

Hmm

This is probably a fairly hot take but I believe that researchers shoudn't normanize providing this level of detail unless they have been explicitly hired to do so.

The industry is pretty messed up and it sometimes feels like researchers/pen testers/bug hunters are effectively subsidizing and protecting shitty security practises.

> The industry is pretty messed up and it sometimes feels like researchers/pen testers/bug hunters are effectively subsidizing and protecting shitty security practises.

In my opinion, this is a result of there being little/no consequences of being breached due to incompetence. Zyxel isn’t losing anything from this, and has no incentives to be better.

These devices have been owned for awhile now. I thought everyone knew?

https://www.zyxel.com/us/en/support/security_advisories.shtm...

Do a search for "cve zyxel"

CVE short for CaVeat Emptor in that case :D

It's so uncommon to see "owned" spelled with just alphabetics that I had to read the sentence a few times to figure out what you meant :)

Company that makes this clearly is completely incompetent in computer security, nobody should buy their products. Ever.

I worked with WiFi developement 15 years ago. For testing interoperability we used many vendor's access points. Zyxel was always on the lower end quality-wise. So this story does not surprise me, nothing seems to have changed. Or maybe there is a positive element of surprise; they accepted that that's not the way to do it and fixed it in only 2 weeks.

I have a Zyxel switch that I had to disconnect from my network because every once in awhile it would just go nuts - I'd get a crazy amount of traffic and upwards of 80% packet loss across my network. But just for a few minutes, then everything would go back to normal.

Took me a while to narrow it down to the switch, and I'm still not really sure what was happening, but it hasn't come up again since I disconnected it.

Is there any kind of independent organization from which networking device vendors can seek security certification?

E.g., they get to review all hw / sw / firmware, and if appropriate make a public affirmation that they could find no vulnerabilities for that stack?

UL need to step up their game

Most of the time such reviews are not practical; they cost too much money and don't deliver results. Such review industries are almost always all about the rubber stamp and don't end up employing competent reviewers.

See e.g. the Infineon ROCA flaw that FIPS certification didn't catch (even though that code reeked of them being too clever, and should've gone through proper cryptanalysis).

See: flaws in the FIPS YubiKey (specific to that version! The certification requirements introduced vulns!)

See: every audit of every terrible CA ever.

I have been thinking of daisy chaining 4 or 5 firewalls together for perfect security.

To make sure you have at least one exploitable device with a buffer overflow triggered by a malformed TCP-packge? :D

I've been doing something vaguely similar, largely out of laziness, for the past few years. My ISP's router (largely) retains its default settings, then my main router/firewall[1] sits behind that in a double-NAT configuration (treating the ISP router as being on the WAN side). Works well enough in practice - it's never caused me any issues in the ten or so years I've been doing it (port forwarding becomes a two-step process, but I _very_ rarely need to do that, so it's a non-issue for me).

[1] Typically pfSense, though it's just a generic Eero setup for now while I reconfigure some stuff33333333

Is "backdoor" the right term for this? I was under the impression that it implied intent. I.e., a "backdoor" is something added specifically to provide access through abnormal means, not just something that can be used to provide access.

This seems more like a mistake, particularly since they included the password in plaintext.

It's a locked front door, with the key under the flower pot. (But that's too long to write out each time.)

I see your point. While this uses the normal authentication method (user, password), the undisclosed nature of the account makes it a backdoor practically.

A secret way for the manufacturer to get into your device; the term "backdoor" is very appropriate I think. Most manufacturers would not go about it as clumsily as Xyxel, but it's the same thing.

But implicit in your statement that it's "a secret way for the manufacturer to get into your device" is the claim that Zyxel intended it as a bypass.

It seems like without intent, anything can be called a "backdoor". My browser receives automatic updates, and Mozilla could push an update that gives them remote access to my user account. But I don't think anyone would say Mozilla has "backdoored" Firefox.

Firstly, I think it's a lot to ask that we believe Zyxel put a backdoor in their products and then just forgot about it.

But even if we choose to believe that, it's not wrong to say that it was secret. "Secret" typically means "without the knowledge of others". If I forget the secret combination to my safe, it doesn't become "not a secret" because of that.

In your example, Mozilla has presumably made public that they are using remote updates. So it can hardly be described as secret.

Not at all: https://en.wikipedia.org/wiki/Backdoor_(computing)

By the looks of it, yes you're right.

This is not a backdoor, this is frontdoor, plain and simple.

I thought they had found vulnerabilities on my 14,400 baud modem. Phew. :-p

The modem brands I still remember are Zyxel and US Robotics.

They made slightly faster modems at 16800 or 19200 if both sides are running Zyxels.

In short - zyfwp is another way to login as root.

Sigh. I'll never buy another ZyXEL product. What are decent alternatives in a similar price range?

You have some couple year old computer sitting around collecting dust? pfSense, OpnSense, VyOS, etc; there's Sophos XG home edition for personal use that I've had success with.

You want a ready-to-use hardware package? Maybe a UniFi Security Gateway, Netgate (pfSense) appliances fit in the price range.

This is why I use OpenWrt!

No bug bounty?

But this is a backdoor, thus not eligible for bug payment.

Reply on news.ycombinator.com