200,000+ MikroTik routers worldwide compromised to inject cryptojacking malware | Hacker News Reader

200,000+ MikroTik routers worldwide compromised to inject cryptojacking malware

badpackets.net

104 pointspjf75 comments

IMHO Mikrotik are being sloppy by introducing breaking changes to their stable channel. Hence ISPs are reluctant to update automatically, fearing some subtle bridge/VLAN change which is sadly set to happen again (6.43 -> 6.44!). Also doesn't help that the underlying Linux stable kernel updates more than once a week.

Every Internet connected device needs some automatic update functionality by default. It's tricky for routers since you typically do not want any downtime and it's really difficult/expensive (well at least 2x the cost for hardware alone) to do a blue/green (or is it red/black?) deployment. Not to mention since Mikrotik is low end, there is no state replication functionality between routers.

Here is a config for Mikrotik that updates my non-ISP hardware once a week at 3AM: https://gist.github.com/kaihendry/59a656c3883450d2df2fd52574...

And when the ISP opts out of the suggested automatic update default, they need to make the commitment to test and roll up updates in a timely fashion. As we all know this is hugely expensive, and I strongly believe a vendor/"computer program" should be able to provide this service. Customers with these ISPs who didn't update are probably seeing some crazy packetloss.

So the future I'd like to see is Mikrotik updates being automatic, staged to some degree & ultimately non-breaking. Cons are downtime for some & I guess allowing your device to be remotely controlled by Latvians. ;)

AFAIK, most Mikrotik employees speak native Russian and only passable Latvian. "Remotely controlled by Russians living in Latvia".

That's patently false, and also irrelevant.

Are these changes coming in bugfix channel as well? I'm currently in holding pattern going from 6.40.9 to 6.42.9 due to master > slave to bridge change which occurred in March I believe. Point is bugfix seems to be less affected by major changes like this (first I have experienced in 3 years).

tbh I think you should be good with bugfix. I've not tried, just going on what others have said to me.

I'm a stable updating kindof guy.

I would use the built in install function to automatically reboot after the update is downloaded. Your current schedule will reboot the router even if there is no update available.

I have a script I have been running for a year without issue. It check for an update, and then if there is an update installs it. https://gist.github.com/tagno25/b207786b02b89700860028e4e247...

Underlying Linux kernel updates? I thought their OS was built from scratch?

RouterOS is a proprietary set of programs (all the way from an init replacement) on top of Linux (a super old version of Linux at that, Kernel 3.x from 2012 IIRC) with a bunch of patches to hack in all the necessary hardware support.

So ... which SoHo router manufacturer can we actually trust? It seems pretty common in this industry to either not supply security updates, or to only supply them for a very short amount of time.

Seems like the solution is still to see what router models are supported by your favorite aftermarket firmware.

Turris omnia. Open source (both hardware and software), lets you ssh into it directly out of the box (well, after you have set a root password in the gui)

They provide regular software updates, and it is imho a good hacker/tinkerer router.

I agree - got one, quite happy with it. But it's targeted for home use, not sure how well it fits into larger networks (small offices). MikroTik seems like a better fit for that, not sure.

One major advantage of Turris Omnia is that CZ.NIC is a non-profit, so they are not as constrained by profitability of this project, and it's a part of a larger strategy (e.g. you may allow reporting statistics back to turris: https://project.turris.cz/en/global-stats/).

300€ is about 5 times what I would consider a price to pay for a home router though.

this depends on what you want out of it I guess. I wanted a tinker router, one which doesnt treat me like an absolute idiot, with a spf for fiber, can do gigabit easily and is expandable.

The hAP-ac I have also ticks most of these boxes[1], but costs less than half what the Turris does (no PCIe, but I am not sure what I'd use that for in a router).

1: https://mikrotik.com/product/RB962UiGS-5HacT2HnT

This is about 5 times the price of hAP ac from Mikrotik, and hAP ac can do some things much faster due to hardware acceleration of some of its capabilities in Qualcomm QCA8337 switch chip.

See my post above: APU2 by pcengines. Put debian on it.

It's a great piece of hardware, of which I've used dozens for various purposes. But they draw ~10 times more power than, say, a cheap Linksys wifi router. So if all you're after is a SoHo wifi router, it might not be the best way to go.

They don't. Mine draw under 10W, which is well within range of a Linksys thing.

Mine draw around 10W idle, about 15W when active. My Linksys and TP-Link trinkets draw under 2W idle (which I'm guessing is my power meter's minimum value), and not much more when active.

Power meters generally don't well measuring low loads, 2W sounds like an error.

That's fair, but I don't know of any SOHO routers I trust, which is what led me to it. It's a tradeoff, but the same tradeoff being made by these mikrotik users.

There is this one the horizon though: https://up-shop.org/up-ai-edge/233-up-net-plus.html

It'd make a fine router in the same vein, no word on power usage though.

I have one running pfSense. It's never given me any trouble.

Germany (or rather DACH) has AVM, a German manufacturer that is making the "Fritz!Box" product line and supplied security updates to all affected routers they ever sold after somebody discovered and exploited a bug in the firmware to remotely call premium numbers via VoIP in various countries.

Their routers were also not affected by the KRACK WPA2 exploit last year.

AVM products cost a lot more than their competitors' but they are really worth their money.

I can agree with that, AVM Fritz!Box are also quite capable routers, the only problem I ever had was that it doesn't properly forward IPv6 ICMP.

It's 100% worth the extra cash getting an AVM over most ISP routers you get in germany.

Not affected by Krack? Is this the marketing mouth of AVM?

As KRACK is a client vulnerability, an access point shouldn't be vulnerable in the first place.

Mikrotik is trustworthy enough. The update that fixed this was released in April, the first exploit appeared a few months later but this story gets regurgitated every couple of weeks because so many people use insecure configurations and don't bother upgrading their routers.

As to their update availability - I'm fairly certain I could take something they manufactured 15 years ago and update it to the current version, as it all uses the same OS, and they still support every architecture they've used.

Also, while Mikrotik security practices are not great (as I recall, before this vulnerability, admin passwords were stored in plain text), by default, I think zero services listen on the WAN port. Owners must specifically configure their device in a not recommended manner (exposing the low-quality web admin interface to the world) for this vulnerability. It's like exposing your printer web interface to the world and expecting it not to get hacked.

The main problem with Mikrotik is that the configuration interface is too low-level and it's easy to fall into such traps.

I wonder how networking equipment manufacturers can motivate regular users to update their equipment more regularly? Auto update would seems like a logical one but plenty of people have reasons to not update and don't want there router going down at times they cannot control.

For most people router is something they don't want to touch and sometimes they don't even have access to it because it's managed by somebody else.

But as the first comment says - introducing breaking changes is not acceptable and they should appear only in major releases. And then those security bugfixes should be backported on all major versions which were released at least 5 years back. Yesterday I upgraded mikrotik from 6.14 to 6.42 and it took me 30minutes of additional configuration to make everything working again.

Also mikrotik collets a lot of network stats so implementing an algorithm which would restart the router when there's usually the least amount of traffic should be feasible - those updates take less than a minute so it's not like windows 10.

I bought a mini PC with 4 Gigabit Ethernet ports and Wifi, i5, 4G RAM, 32G SSD (probably way overkill for a router but I can do other stuff on it) on Aliexpress for about 250 USD and put Debian on it. I heard it works well with pfsense too.

Try OPNsense, it has hardened GUI server, and is stable as hell.

Ubiquiti makes great little boxes, Edgerouter Lite, Edgerouter POE and Edgerouter 4 have all served me well over the years.

They have hardware offloading for routing and iptables, will route 900Mbps, and get regular software updates.

Edgerouter Lite is $99, and draws about 5W.

Their wired hardware is okay but some is subject to some pretty embarrassing bugs that Ubiquiti is extremely slow to fix [0] and in my own experience, their support is utterly abysmal.

I would recommend buying something else.

0: https://community.ubnt.com/t5/EdgeRouter/EdgeRouter-X-SFP-le... (reported 2017-03, still alive as of 2018-08)

https://blog.mikrotik.com/security/new-exploit-for-mikrotik-...

"Regardless of version used, all RouterOS versions that have the default firewall enabled, are not vulnerable"

I don't believe all MK devices OOTB had the WAN interface firewall'd (they do now though their wAP's run same license level 4 of RouterOS and do not have fw enabled on the ethernet port) though I do recall that being made very clear in both the documentation that came with the hardware and in any wiki/docs I looked at at the time I was first setting up my MK device.

The issue I have is the lack of communication with customers. I have not received any notification of this vuln since August 5th (well since patched in April) but they in general seem rather blase about the whole thing. I haven't followed the link to story but I follow BPR on Twitter and saw that it's around 421K according to censysio.

https://twitter.com/bad_packets/status/1050533001824595968

They've sent emails about this, posted to Twitter and Facebook, the software shows that there is an update available. If you've bought their device from a third party, and they don't have your contact information, how do you expect them to communicate with you?

Purchased everything on Amazon that was sold by Mikrotik. I follow a couple of their accounts on Twitter and will see routerOS version updates there well before I ever get an email. Like I said, I have not received any email about this vuln since August 5th (patched day of back in April). Email notification is paramount for me...

Edit: I see...*notification via email is what I thought I had written in top post.

> A patch was issued earlier this year by MikroTik, however the latest statistics (above) reveal device owners and network operators have chosen not to apply it.

So MikroTik did its part.

I gave up on all-in-one firewall/router/wireless AP units recently and now have a more complex setup I'd only recommend to those who enjoy setting these things up and learning.

For the firewall I've got pfSense (FreeBSD based firewall) running on an small Intel box from Aliexpress with 6x ethernet ports. I overspecced the machine so I knew it would last for years, whilst allowing running intrusion detection (snort), reverse proxy, auto blacklisting and much more.

This is also acting as my router, but if & when my home network expands I'll add a dedicated one (and probably put 4x ethernet ports bonded using lagg into it as it'll still be controlling my vlans).

I got a dedicated ceiling mount wireless AP from TP-Link.

This has only slightly higher power and space requirements than what I had, but that buys me so much flexibility. I can upgrade and augment individual components when needed (particularly wifi). I have individual vlans, firewalled off from each other, running over three wifi SSIDs - trusted, guest and internet-of-shit. I'm currently connected into my trusted vlan over OpenVPN, so I can connect to my server without having to open any ports to the outside world.

The downsides are that it did take a couple of days to set it all up to my liking, but personally I've learned so much doing it it's been worth every minute.

You only need to trust them for the hardware. Then install openwrt.

http://pcengines.ch/ APU2 plus debian. Be secure. Maybe I'll tidy up the ansible I use for this and publish it.

Please do !

Or *BSD. Great device.

So those don't need updates?

If you have a currency that can be generated via compute power the incentive structure it creates is to take over as much compute power as you can. Does the incentive structure represent a serious design flaw in the system for widespread adoption?

I would say it's a consequence of computer power becoming commoditized - the flaw might be that using computer power in the past can be 'stored', unlike hijacking someone else's computer and renting it out as server space, there's a point in this kind of crime where one has the profits and got away with it.

I just had this thought and want to get it down, apologies for the rant: I suspect that philosophically, the basic measure of our economy has been units of energy, but it's transitioning to units of power. The 'subscription model' most businesses are transitioning to seems to be an early recognition of that fact. Server space, similarly, isn't really something one can store as easily as gold or gasoline - it's only valuable while in use and degrading whether it's in use or not. Our whole concept of currency may be based on an idea of 'stored' value, analogous to swapping joules, while we should be thinking about 'sources' of value - swapping watts. I think the trouble is that renting (or hijacking) servers is accounting in terms of watts, while currency is accounting in terms of joules. The accounting of the two types of approaches may not be as easily reconcilable as it seems at firs.

I think both types of currency are important. Humans have basic needs which need to be met in the shape of atoms and joules which are quite storable. We also have luxury wants which translate better to watts. This separation might also be useful in the context of UBI schemes. And more generally I have a gut feeling, but can't quite substantiate, that many fairness problems stem from using the same currency for both, forcing the lower classes to work multiple jobs just to make basic needs while the haves compete with them just to improve their lifestyle and status.

Fascinating! That makes excellent sense.

this is getting even bigger: 421K+ routers compromised - https://twitter.com/bad_packets/status/1050533001824595968

Ha!

https://www.zdnet.com/article/a-mysterious-grey-hat-is-patch...

Does anyone know if Ubiquiti's edgerouters are any good?

They are. They run a fork of Vyos (https://vyos.io/), and get regular updates.

In my experience, no, they're not. Their wireless gear is great but when it comes to wired, I'd avoid them. There's a serious bug [0] on the ER-X-SFP that's been around for over a year without any serious action from Ubiquiti. Their support when I tried to get an RMA for said issue was utterly abysmal.

[0]: https://community.ubnt.com/t5/EdgeRouter/EdgeRouter-X-SFP-le...

Agreed, I run MikroTik on the wired side and UniFi for the AP side; have many installs in my charge and this is our default deployment when “more WiFi” is requested.

I have an EdgeRouter X and I'm searching for a replacement.

EdgeRouter X issues:

* 3.10 kernel. This is out of LTS support.

* Poor IPv6 support. The GUI has practically 0 support and you instead have to learn EdgeOS config, and it's awkward.

I'm happy to use EdgeRouter X as a switch, so I'm looking for a SBC that can act as router/firewall and run vanilla Debian.

Scratch that. Upon reading VyOS's docs for the EdgeOS config (which has relatively poorer docs), it has strong advantages: commit/rollback, single config file, and Ansible has an EdgeOS module included by default to coordinate that.

The latest development release updates the ER-X to kernel 4.4, which is LTS until at least 2022. The base OS is updated to Debian Stretch also.

I think that's a pretty good level of support for a $49 device.

I guess nobody listened [0].

But considering some of the comments on that thread, I'm not entirely shocked.

> Attackers will not be able to use that, nor will they care.

> If you're exposing that unfettered to the web at large, imo you deserve what you get.

Did no one really think 200,000 devices was a worthwhile target?

[0] https://news.ycombinator.com/item?id=18166003

I have several hAP ac Mikrotik routers and upgrading them is a pain. You can not just download an image from their website, flash and reboot. If you do so, your router will likely be locked in a bootloop.

I managed to have consistent upgrades by using only the main package and Netinstall, but it is still a huge pain in the ass.

Mikrotik makes stable routers, but they messed up the upgrade process completely.

Huh? With the exception of them converting all master > slave port configs to bridges in 6.41 I believe, I have never had any issues using System > Packages > Check for updates, selecting bugfix as opposed to current branch and downloading and installing in Winbox. SwOS devices I need to download a binary and upgrade through web (no Winbox) but still have never had any issues.

Try doing it through the web interface, you'll be unpleasantly surprised.

I just upgraded my last hAP ac from 6.39.2 to 6.42.9 through the web interface, entered the bootloop, then did the Netinstall of the system package only, then manually restored the configuration.

I exclusively use the web interface and haven't seen this after thousands of upgrades.

Upload the package to the router(make sure it completes)

Reboot the router.

Upgrade the bios "Routerboard firmware"

Reboot the router.

And done.

This is why we do backups :) I always assume something will go wrong, but make sure to have a backup of any critical device that is being updated (mikrotik or not).

Did your hAP run out of flash disk space? I notice that it only has 16MB.

I recently purchased their outdoor wAP ac (same amount of flash) that deletes any "backups/exports" on disk after reboot. I'm not familiar with CAPsMAN but I wonder if there's a mechanism to backup to either the controller or external repo. There's always the scripting route to ship configs elsewhere etc.

I email the backups to myself (routerOS scripting) using a dedicated gmail account, but the backup is also written to an external USB stick on my device.

I don't see it any more cumbersome than Juniper, Cisco, or any other enterprise hardware manufacturer(not saying that Mikrotik is enterprise hardware...) I manage several networks totaling 500+ Mikrotik routers.

Once or twice I have run into upgrade issues that had to be resolved with a technician on site (broke vlan configs)

Just once the update wiped the config.

This is out of several thousand successful updates. I've run into similar issues with other big router manufacturers.

Like most companies, you do not want to be on the 'bleeding edge' of their updates unless absolutely necessary. They do have a significant problem with regressions in their updates.

I also have several hAP ac routers and had no issue upgrading them. I only had a problem with an older routerboard (first time I tried upgrading my first ever Mikrotik), but that was because I did not know how the upgrade works and I cut off the power during the reboot (I was under the impression I have to reboot it manually).

I never use the web interface though - winbox is way better.

What? I have updated all my MikrotikDevices by literally just dropping the new firmware image onto the device and restarting the device. That's it.

It takes literally two clicks on their website to download an image for whatever version of their OS you need.

Yes, the web interface for automatic updates (where the router is responsible for downloading the firmware) is very poor at updates. However, who uses that in the first place?

The only method I have ever used is dropping the .npk packages into the root of the filesystem via sftp or ftp, and rebooting. No Netinstall required.

I've done this something like 50 times, on multiple models covering multiple architectures (Mips, PPC, x86, Arm), and never once had any issues.

Uploading them one-by-one via the web interface or Winbox (via the Files section) and rebooting also works just fine, but why bother when you can do them in one shot as above?

And if you have to upgrade a large number of routers, best of all is write your own script that pulls firmware from your own private repository on your management VLAN, and reboots. And seriously, anyone who is managing a large number of routers and is not bothering to test image upgrades in a testing environment before deploying to live deserves any trouble they get.

Only issue I sometimes had with updates is limited memory (on the lowend hap lite devices). Often a reboot before upgrade download was enough to solve that.

But the standard upgrade procedure is to put the new firmware image in the filesystem root (scp, update button via UI, etc) and reboot the device.

I'm really curious, where are these routers? The problem is that the admin interface port is exposed to the internet, which is something any competent administrator would ensure isn't accessible. So are there incompetent admins managing 200k devices or is someone distributing these to residential users?

When I first started reading about these reports over the past week I believe around 90%+ were in Brazil. If you peruse BPR timeline on Twitter they mention type of orgs using them. One of hardest hit I believe was ISP in Arizona or New Mexico, US.

It's good to regularly update your router.

Reply on news.ycombinator.com