Hacker News Reader

Comment by shoddydoordesk | Hacker News Reader

> it suddenly ballooned in size in April 2025 after its operators breached a TotoLink router firmware update server and infected approximately 100,000 devices

This is scary. Everyone lauds open source projects like OpenWRT but... who is watching their servers?

I imagine you can't run an army of security people on donations and a shoestring budget. Does OpenWRT use digital signing to mitigate this?

View on HN

Why, OpenWRT firmware and packages are both signed, of course. You can manually and independently check the image signature before flashing an update.

The build infrastructure is, of course, a juicy target: infect the artifact after building but before signing, and pwn millions of boxes before this is detected.

This is why bit-perfect reproducible builds are so important. OpenWRT in particular have that: https://openwrt.org/docs/guide-developer/security#reproducib...

Bit-Reproducible infrastructure could also result in some of the wildest build distribution architectures if you think about it. You could publish sources and have people register like in APT mirrors to provide builds, and at the end of the day, the build from the largest bit-equal group is published.

I do see the Tor-Issue - a botnet or a well-supplied malicious actor could just flood it. And if you flip it - if you'd need agreement about the build output, it could also be poisoned with enough nodes to prevent releases for a critical security issue. I agree, I don't solve all supply chain issues in one comment :)

But that in turn could be helped with reputation. Maybe a node needs to supply 6 months of perfect builds - for testing as well - to become eligible. Which would be defeated by patience, but what isn't? It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer.

This combination of reproducible, deterministic builds, tests across a number of probably-trustworthy sources is quite interesting, as it allows very heavy decentralization. I could just run an old laptop or two here to support. And then come compromise hundreds of these all across the world.

The distribution system you're describing exists and has been in use for decades. You just distribute the build using bittorrent.

And if someone invests in having >90% of the peers offer a malicious file and serve DHTs matching that file?

Torrent files are hashed, so it's exactly the same risk profile as the comment I was referring to. But generally hashing algorithms are collision-proof enough that what you're describing is basically impossible (requiring many years of compute time).

IIRC BitTorrent still uses SHA-1, which is becoming more problematic.

BitTorrent v2 uses SHA-256, but in any case SHA-1 is still second-preimage resistant. And the BitTorrent piece hashes are included in the .torrent file, so you would need to find a double collision.

>It'd just have to be more annoying to breach the distributed build infrastructure than to plant a malicious developer.

It really wouldn't. You don't even need a powerful build server since you can mirror whatever someone else built. You can also buy / hack nodes of existing trusted people.

Sounds overly complex and completely unnecessary, like some kind of blockchain/defi scheme shoehorned onto distributed builds.

> Try building some common pieces of software in a network isolated environment and you will likely be surprised at how poorly it goes.

I have yet to experience a straight shot install or build of anything in an air gapped environment. Always need to hack things to make it work.

Reproducible isn't quite enough, you also need bootstrap from almost-zero binaries.

https://bootstrappable.org/

This exchange is somewhat hilarious. Oh how on earth do we keep things safe and secure if everyone can see the code and verify what it does! Who would keep us safe if we turn our backs to unverifiable, unvetted, unprofitable security fixes, by for-profit companies!

> how on earth do we keep things safe and secure if everyone can see the code and verify what it does!

That's not always the silver bullet you seem to think it is. Have you ever tried to build something like Chromium, Firefox, or LLVM yourself? It's not realistic to do that on a mid tier let alone low end device.

Even when you go to the trouble of getting a local build set up, more often than not the build system immediately attempts to download opaque binary blobs of uncertain provenance. Try building some common pieces of software in a network isolated environment and you will likely be surprised at how poorly it goes.

If projects actually took this stuff seriously then you'd be able to bootstrap from a sectorlisp and pure human readable source code without any binary blobs or network access involved. Instead we have the abomination that is npm.

Debian manages to build Chromium, Firefox, and LLVM on servers of multiple architectures, including quite slow riscv64 machines, without any network access to the builds for any architecture.

https://buildd.debian.org/status/package.php?p=firefox-esr

See Bootstrappable Builds for starting from almost nothing, so far only GNU Guix and StageX have worked out how to start from the BB work to get a full distro. Should be fairly trivial for other distros too if they cared.

https://bootstrappable.org/ https://guix.gnu.org/blog/2023/the-full-source-bootstrap-bui... https://stagex.tools/

For context, I once found a bug in Chromium and fixed it, the initial build took a few days on and off on my development laptop that was pretty beefy for the time. I say on and off because I had to interrupt the build if I wanted to do anything else computationally taxing. They have incremental builds and caches all properly set up so you can just continue where you left off after the fact. After the initial build it's pretty fast, 5 minutes or so per build for me. On a low end device you're easily looking at a build time of a week or more if you're starting from scratch.

LLVM isn't so bad compared to the browsers. Relatively standard CMake build with mostly self contained c++ codebase and few third party dependencies. You don't need a crazy thread ripper workstation to do a build in reasonable time. A somewhat modern 8-16 core desktop CPU should be able to do it in 10-20 minutes or faster. Based on compilation benchmarks I have seen even some of 15 year old 4 core CPUs or 5year old mid/low tier mobile CPUs do it under hour.

Most importantly you need to pay attention to RAM usage, if necessary reducing parallelism so that it doesn't need to swap.

The biggest joke is most of the proprietary routers both consumer and enterprise grade often are running some old outdated version of custom tuned openwrt lol, this goes for tp-link, and everyone else almost.

> You can manually and independently check the image signature before flashing an update.

Of course you can. You can also read the ToS before clicking accept, but who does that?

I'm sure there are dozens of us.

Ever since that one game with the soul-surrendering clause in the EULA, I read it all now, heh.

People who don't want to find themselves inadvertently participating in a botnet.

I don't follow.

> run an army of security people

Do you think these private companies do this? They don't. They pay as little as humanly possible to cover their ass.

Botnets comprised of compromised routers is common and commercial/consumer routers are a far juicer target than openwrt.

> They pay as little as humanly possible to cover their ass.

They probably spend more on the team who ends up writing the "We take your security very seriously" breach notification message than they do on "security people". At least until then get forced into brand-name external Cyber Security Consultants to "investigate" their breach and work out who they can plausibly blame it on that's not part of the C suite.

> They pay as little as humanly possible to cover their ass.

It’s probably helpful that open source teams aren’t hampered by standards and 20 year outdated audit processes either.

This is exactly why OpenWRT has no unattended updates by default )

You are dismissing the seriousness of this. Their package manager is widely used. One would only need to compromise their build servers to wreak havoc.

Didn't they have a vulnerability in their firmware download tool like a minute ago?

The difference between OpenWRT and Linux distros is the amount of testing and visibility. OpenWRT is loaded on to residential devices and forgotten about, it doesn't have professional sysadmins babysitting it 24/7.

Remember the xz backdoor was only discovered because some autist at Microsoft noticed a microsecond difference in performance testing.

I'm confused why you're so honed in on OpenWRT as a third-party open-source project here when the vulnerability you quoted (TotoLink) was the official firmware update server of a brand of devices.

Is it "scary" to think about OpenWRT potentially getting hacked? If you get scared by theoretical possibilities in software, sure. Is it relevant? Not exactly. Are companies' official servers more secure than an open-source project's servers? In this case, apparently not.

What's scary is that OpenWRT is a project created by people who wanted a better solution than what was out there, and are therefore largely driven by a desire to create a good product.

Meanwhile, corporations are driven entirely by profit motive, so as long as it's more expensive to be vigilant about security than it is to be lax about it they will never improve.

Until companies which produce (and do not update) vulnerable equipment are penalized (e.g. charged with criminal negligence) for DDoS attacks using their hardware then the open-source projects are going to continue to be far more trustworthy and less vulnerable than corporations which mass-produce the cheapest hardware they can and then designating it as obsolete and unsupported as fast as possible to force more updates.

The disappointing thing is that the companies don't just ship the open source firmware on their devices from the factory. They rarely if ever have any marketable features the open source firmware doesn't -- it's more often the other way around -- and then you don't have a zillion unpatched devices when they decide to stop caring because the community continues to maintain the code.

The post is nothing more than "but what about security" meant to deflect away from the discussion at hand and towards OpenWRT

As always, hundreds watch the open repositories, maybe one watches a company's build servers, if they're lucky. :-)

Hundreds watch, but how closely?

Plenty of stories of fairly major projects having evil commits snuck in that remain for months.

Name a few.

https://en.wikipedia.org/wiki/XZ_Utils_backdoor

https://medium.com/@aleksamajkic/fake-sms-how-deep-does-the-...

https://blog.linuxmint.com/?p=2994

https://www.bleepingcomputer.com/news/linux/malicious-packag...

https://www.cnx-software.com/2021/04/22/phd-students-willful...

I could go on but I trust this is a sufficient number of examples.

Only two of these were actual malicious commits. Two others were malware inserted into the repositories (if Twitter could be thought of as a meta-repo), which is bad but not on the same scale.

I wonder why nowhere talked about who Jia Tan was. In my understanding, a few people already talked to that person. Now, does Jia Tan really vanish?

I recently had some issues getting one of our embeded devices connect through passive ftp. Because the exact same device worked at a different site I knew it wasn't the device or it's settings. Long story short, it turned out the problematic site hadn't been updating its routers which meant they couldn't VPN passive FTP traffic. Anyway, we have literal thousands of those routers maintained by hundreds of different companies, who are mainly there to maintain the actual mechanical equipment and not the network. Turned out the site where the technicians updated things weren't in the majority.

I'm in the process of getting the business to implement better security, and it's going better than you might expect. If it wasn't because having a plan for how to update your OT security is required to meet EU compliance, however, I doubt we would've done anything beyond making sure we could do passive FTP when it was needed.

As an example, there is still no plans to deal with the OT which we know has build in hardware backdoors from the manufactures. Wnich is around 70% of our dataloggers, but the EU has no compliance rules on that...

Digital signing wouldn't defend you from a compromised build server.

What in that act says OpenWrt would be made illegal? If anything, OpenWrt would roll out automated security updates for a supported branched release to comply with these regulations.

Also, if you actually read it, there are exceptions for open source software!

OP claims almost daily that some benign thing is actually illegal but practically never provides any useful proof when asked.

(please prove me wrong, Alex)

Reproducible Builds and multiple distributed builders would though.

https://reproducible-builds.org/

Reply on news.ycombinator.com