Comment by JdeBP | Hacker News Reader

Comment by JdeBP | Hacker News Reader

I don't have an ASUS router, but three things leap out at me: a string being logged over and over, running out of space on a filesystem, and rotated log files named something.1.

It is trivially easy to blow right past the size capping on systems that use the old "newsyslog" style of external logfile rotation from the 20th century, and something that is logging a short string "[chknvram_action] Invalid string" over and over very fast is exactly how to do this.

For those interested in investigation, therefore, I would suggest looking at logfile sizes, and seeing whether it was logs eating all of the free space on /jffs and /var .

The underlying cause would be whatever is logging "[chknvram_action] Invalid string" thousands of times over, but the mechanism would be log files filling the tmpfs that the article mentions, which would explain why the system had no memory for forking new processes.

My wild speculation about "[chknvram_action] Invalid string" is that something somewhere in whatever "chknvram" is, the name being suggestive of something checking non-volatile RAM, has either bad data or a broken parser, and the recovery semantics are to retry immediately, incessantly, as fast as possible.

It turns out that chknvram is one part of the asd service, and it regularly downloads signature files looking for malwares.

* https://www.snbforums.com/threads/what-is-asd-process.76242/...

So some somewhat more informed speculation is that the new signature file either yesterday or today either broke a parser or was itself corrupt. The error-handling path for this is still poor.

In the comments of the article someone mentions that deleting the file solved the issue without a firmware update. Too bad they didn't save it before, a comparison with the newer working version would be nice.

I guess Asus quickly discovered their mistake and removed the faulty file from their servers, but affected devices never got to the point where they'd look for a newer file but just choked on the local one.

> it regularly downloads signature files

Why does a router need malware signature files? It has no business monitoring my traffic, except in accordance with the firewall rules that I set myself.

From the article:

> not keeping my firmware up to date

I've had this (non-Asus) router for three years. I've never updated the firmware.

> I've had this (non-Asus) router for three years. I've never updated the firmware.

Botnet operators are very grateful for your cooperation, you are helping them a lot

I've never updated the firmware on any of my routers, and I'll be happy to give you a million dollars if you can prove that it's part of a botnet. This pathethic holier than thou attitude is obnoxious, you really think 95% of routers are part of a botnet? Or that they're even accessible from the internet?

As 100% of Asus routers are part of the Asus botnet so I'll take my chances on that 5% with my custom, libre, hand-rolled breadboard router.

> you really think 95% of routers are part of a botnet? Or that they're even accessible from the internet?

It's probably not 95%, but it's certainly significant. And yes, most consumer routers are obviously accessible from the internet, that's how they work (they're generally not behind a firewall, they are a firewall, and can have vulnerabilities). You might want to read up on it [1, key quotes below], and I know I certainly wouldn't be willing to wager my consumer router wasn't infected, because how would I ever have any idea?

> Out of the box, most routers suck when it comes to security. Vulnerable firmware is an easy target. Backdoors have been discovered in just about every brand of router... It's possible you will never even know your device is infected...

> When we're talking about routers, these aren't the enterprise-level gear made by the likes of Cisco, Sonicwall, or Palo Alto... We're talking about what you would get off the shelf at Wal-Mart or Best Buy — the D-Links, Netgears, and Linksyses of the world.

> The problems start with the cheap, antiquated MIPS architecture used by a lot of these devices' processors... One substantial problem comes from a security flaw that goes back to 2001. Per one deep yet fascinating paper, the chips lack basic defensive abilities against malicious code execution. Combine this with manufactures commonly implementing out-of-date, vulnerable Linux kernels in their devices, and then putting this device on the edge of your network completely exposed to the internet.

But to sum up -- not updating your router's firmware is a terrible idea. It's not something to be proud of.

[1] https://www.cbtnuggets.com/blog/certifications/security/your...

If you don't expose any external services of the router then it's a bit hard for botnets to take advantage of your router.

Most of these routers are likely becoming part of the botnet for enabling external web management and/or using default creds (especially if SSH is exposed externally).

Security isn't exceptionally hard, if you actually put some effort into it.

There have been plenty of router exploits that begin with a pivot from a web browser to a poorly secured admin panel on the local network. Firewalling incoming traffic from the Internet to the router's management interface is no security panacea.

Again, as I mentioned, the bulk of those rely on default creds.

Many of the big manufacturers have been tackling that issue by forcing a password change at setup and not allowing an insecure default to be chosen.

> a poorly secured admin panel on the local network

Why is the web interface left on? Just turn that off, there shouldn't be much to do with that.

Not sure you'll see this, with HN's lack of notifications, and so much time elapsed but here's the answer:

Most consumer routers do not support disabling the web interface on the local network, as it's the primary (only?) means of administration for them. This attack relies on getting users to browse to an address with default creds by some means, with a URL prepended that will cause the desired action to occur. More often than not a popular action is to modify the DNS servers used, so that DNS traffic can then me manipulated to point to malicious servers used for the ultimate attack.

Don't forget the biggest botnet being ASUS who didn't disclose the scanning... or whatever else they happen to be doing without the explicit consent of the owner.

Issues like this don't happen across such a wide area by accident.

Basic professional practices have any updates tested beforehand on physical hardware at multiple stages before any push happens, and they leave it up to the user because pushing an update without the owner's explicit consent to a device they own, runs afoul of the same hacking abuse laws. The legal exposure is massive.

It also doesn't account for the fact that firmware was attempted to be updated on devices which were set to not update. ASUS has a lot to explain. They didn't release any kind of statement so the lawyers and computer forensics people will likely need to get involved to get to the bottom of it.

I look forward to eventually hearing about what really happened.

Don't think so. The management interfaces are only exposed on the local network. If you could hack into my local network some other way (e.g. through Apache), you could probably attack the web interface on the router; but in fact no local network services are exposed to the internet.

It's a device that's directly exposed on the public Internet. Anyone in the world can send any package they want to it. It's not that uncommon for people to find bad bugs in network protocol stacks. An attacker wouldn't have to go through a web interface.

There are a multitude of ways to trick consumer routers into letting an external request log into the web interface. That's also ignoring the multitude of ways to botnet them without using the web portal, as manufacturers suck at their job and often ship known exploited libraries and utilities. Some have had genuine back doors even.

Botnet operators are careful to not mess with my internet connection. If they want to use my router for other stuff while it’s not in use by me, who am I to complain.

Well, for starters here is an article about malware specifically targeting ASUS routers.

https://www.trendmicro.com/en_us/research/22/c/cyclops-blink...

Hopefully that clears up the first question of "why does it need malware signature files?"

As for your router firmware? You should seriously update that. New exploits get found all the time.

Hackers use compromised routers as parts of a botnet, a intermediary route, or as an access point with which to steal data with Man in the Middle attacks.

That’s an attack on the router itself, not the network traffic it carries.

Signature files are only useful to scan network traffic.

This was literally a discussion about the firmware of a router.

How is it not relevant? This person has not updated their router's firmware in over 3 years.

The viral traffic needs to get to the router in the first place. I assume that the means of reaching the router is literally via network traffic?

What am I missing?

> Hopefully that clears up the first question of "why does it need malware signature files?"

The malware signature files really don't help prevent your router joining a botnet.

Firmware updates, maybe, maybe not. It is quite possible for other routers with less generally sloppy and advertised-feature-rich firmware to actually be more secure even without updates for 3 years. It's quite possible that they have no api endpoints available for super-easy mobile app integration remote management etc, just ssh from local subnet or physical serial console.

There have been multiple cases of market-leading antivirus engines (symanted, mcaffee, etc) having sloppy code running with the highest possible privilege, parsing any files appearing on the system anywhere, and e.g. crashing a mail server that would otherwise be unaffected by the PoC samples being emailed through it by researchers.

So, I also take some issue with people who have no understanding of how all this software around us is designed and built (in routers, in windows, on web servers) and thinking that just updating everything all the time and running antivirus is the best you can do. You really can do a lot better if you know what you're doing.

Re: firmware updates, there is stuff like these remotely exploitable kernel wifi stack issues not that long ago:

https://lwn.net/ml/oss-security/20221013101046.GB20615@suse....

There can be driver specific remotely exploitable issues that might not be widely communicated. Until operating systems are written more robustly, just having admin level stuff set up robustly isn't always enough. Of course, updates can add bugs too.

When I hear “signature file” I think of a list of signatures of known viruses and malware.

These types of signature files aren’t meant to guard against exploits, SSH brute forcing, etc, even if the router applies them to inbound traffic in addition to forwarded traffic. To do that, you typically need a WAF or some clever fail2ban-like filtering rules. Even up-to-date signatures won’t prevent a router from getting 0wn3d if the ssh daemon has a security hole for example.

As sites move to HTTPS, routers can’t even really filter networking traffic anymore. I don’t see why a router needs signature lists at all

Thankyou for the insight.

> Hopefully that clears up the first question of "why does it need malware signature files?"

Not really. The article doesn't mention signature files. It describes the operations of a certain malware once it has been executed on a router. But first it has to get onto the router; and the only way new software can get onto this router is via a firmware update.

>the only way new software can get onto this router is via a firmware update.

That has never been the case. Software from consumer routers is often still in the "a trivial buffer overflow lets a malformed packet insert a payload into ram and convince the PC to jump to it." phase of software security. They are very much still wormable systems, like Windows 2000 style. Ever have your router glitch out and stop working and you have to reboot it? That's probably an exploitable bug.

Love that thread. In typical SNB fashion everybody piles on OP for wanting to disable the scan that keeps his drives awake for days on end.

They argue that he doesn't understand and he's stupid to want to be part of a botnet and that Asus obviously know what they're doing.

An extra fun part is that about 2/3rds of the way through in pops someone else pointing out that the closed source ASUS proprietary malware scanner decided that all of xyr /usr/lib/crt*.o files on a mounted disc volume were malwares and promptly deleted them. (-:

Why are people mounting discs?

Probably they want to share files or to install extra software on the router?

Why not just put the drive on the network? I'm getting downvoted but I'm just trying to understand how a router is able to delete things on an external disc.

Some routers can also double as a budget NAS by plugging a USB hard drive into it. I've seen some that also double as print servers.

How would you put the drive on the network? Attaching it to an always-on computer, or buy a nas device or something? It seems easier to plug it into your router if the firmware supports it.

You can use a lot of routers as low cost NASs.

When I was a broke college student sharing my neighbors open WiFi (with permission) I convinced him to attach a HD to his router. I ripped all our DVDs, Blu-ray’s, and put our torrents on the HD. We could watch videos from our laptops connected to TVs.

The sea gate drive failed soon after and I lost tons and tons of files. Not to mention it silently corrupted almost every file before biting the dust, videos stopped playing suddenly.

Many of the ASUS routers have a built in feature to use a usb drive's space as FTP/SMB share on the network. Can be handy, although slower than an actual NAS.

A parsing error due to a signature-based malware definition file update is a totally plausible suspect!

It would explain why the router is downloading “updates” but not firmware upgrades.

Also, these signature files contain tons of hex strings and unusual characters used to identify the actual malware (IOCs).

We rollback these updates all the time when a bad malware signature update pegs the AV scan daemon. They are released several times per month depending on the vendor.

Someone more knowledgeable about ASUS asd can probably confirm/deny.

The most annoying part is that security scanning is opt-in (I have it off) but I guess the service auto starts and auto downloads definitions anyway.

What's your recommended alternative to the newsyslog style of external logfile rotation? I'm not much of a sysadmin but it might be useful to know at some point. Thanks in advance!

The one that people came up with in the 1990s. There are quite a number of implementations to choose from. The shame of this hitting ASUS in 2023 is that this is a long-known problem and a long-since solved one. I have vague memories of grumpy posts on Usenet about this. It's that old a problem; and it has been solved for nigh on a quarter of a century.

See https://jdebp.uk/FGA/do-not-use-logrotate.html for everything from Bryan Cantrill to comments in GNU source code. (-:

Awesome, thanks! I am in fact guilty of using logrotate but thankfully I haven't been burned yet. Although perhaps with the advent of containers piping logs to streams I've unknowingly absolved myself.

Also, you have an amazing website. Looking forward to reading more.

Weird how syslog doesn't de-duplicate these identical log messages. Or maybe it does, but not enough.

BSD syslogd definitely does, perhaps whatever stripped down one ASUS is using doesn't. e.g.

  Dec  2 01:09:41 hostname syslogd: last message repeated 10 times

The threshold's pretty low and most of the "repeated" messages say "repeated 1 times" however.

Does it store timestamps for those repeated occurrences? I wouldn't want my logs to "helpfully" coalesce multiple identical messages into a single one. For example, if I saw the equivalent of the text below in a program/OS while facing some issues, I'd be remarkably pissed.

  [2023-05-18 07:12:21] [E] Invalid event 0x1AF2 received from ...
  [2023-05-18 09:44:01] [I] Last message repeated 10 times

I care less about how many times the message was repeated - I care about timestamps, which I might want to correlate to other activities.

No, it's just a plain syslogd dating back to the 4.2 days. It aggregates at 30, 120, and 600 second intervals according to the source. Within that threshold I wouldn't care too much. If I really needed timestamps with more than thirty second precision I probably wouldn't be using syslogd.

In any new-ish production system I'd probably want to use anything other than syslogd anyways.

https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/sy...

That's... good to know. I never realized anyone is doing something like this, ever. It breaks my trust in software logs in general - I'll be sure from now on to understand how any given program handles logging, before making assumptions relevant to troubleshooting.

In many cases, logs are asynchronous so depending on many factors among which are utilization of the host, you might get them with a delay and the ordering of events might not make sense because of that when read from the logs. If you need that precision you can surely engineer/ configure your system for that.

If I'm running my own distributed system for some business reasons, sure.

If I'm dealing with equipment failures, bugs in third-party software, or other such random tech bullshit, as an individual or a team, then I don't know in advance when and what precision I'll need.

In a reasonable system, you might be able to change some function in one place, perhaps even in the running system and get the precision or detail you need. You might take out the big guns like e.g. dynamic tracing using BPF to attach probes at the right places.

If your logs are being spammed with the same message, while nothing else logworthy is happening, how many timestamps do you need in one hour?

Most of the time, I'd assume all of them I can store (deduplication is fine, if I can recreate the raw data afterwards). Which is a lot, because they should compress well (in the limit, approaching the same size as deduplication solution).

Sometimes those data points don't matter - like if they're generated by some program stuck in an infinite loop. But in other cases, they do - like e.g. if each message is caused by some event, like another program doing some processing, or user pressing a key, etc. - then timestamps will be useful to identify the exact cause (e.g. logs only happen when process X is processing mouse input, or when user presses one of 20 specific keys on their keyboard, or only when my microwave oven is running).

It helps to know exactly at what interval things are occuring, and you lose fidelity with this style of logging.

> BSD syslogd definitely does

…by default, but it can be disabled:

     -c      Disable the compression of repeated instances of the same line
      into a single line of the form "last message repeated N times"
      when the output is a pipe to another program.  If specified
      twice, disable this compression in all cases.

* https://man.freebsd.org/cgi/man.cgi?query=syslogd

Respectfully, logs shouldn't be "smart", they should just log, that's it.

It seems that the binary in question just writes the logfile directly and does not use syslog.

Reply on news.ycombinator.com