Ubiquiti accused of covering up ‘catastrophic’ breach– and it’s not denying it
theverge.com
theverge.com
Whistleblower: Ubiquiti Breach “Catastrophic” - https://news.ycombinator.com/item?id=26638145 - March 2021 (770 comments)
Previously:
Ubiquiti Networks Breach - https://news.ycombinator.com/item?id=25735032 - Jan 2021 (467 comments)
Remember, there were VMs spun up in their cloud that they couldn't account for. That doesn't look like someone just tooling around to me, that looks like someone recreating a build process.
Is anyone monitoring their Ubiquiti devices in their homelabs to watch for any suspicious traffic? Being part of a nation-state botnet is not a very comforting idea.
With modern software good luck filtering suspicious noise from signal.
So, this hack also implies that a substantial number of devices out there now have their remote admin credentials leaked to an unknown, presumed malicious, third party.
I suppose an automatic firmware update with bad code in it could still hit me?
Very hard to say based on the limited information released. Part of what was released mentions "SSO cookie secrets" and "remote access". Best to assume 2FA isn't protecting you if the attackers were possibly able to forge cookies.
The concern is whether their remote login / SSO authentication via cloudkey was compromised.
But 2FA in this case is useless. The hackers owned the UBNT servers. If they harvested user logins, then one can assume that they also harvested the corresponding TOTP secrets.
To make matters worse: When you setup a Gen2, Gen2+, UDM, or UDM-Pro, you are required to login to the UBNT cloud service to activate your device. Once you do this, your device now has an "Owner" account cannot be disabled.
If you have enabled 2FA on your Unifi account, then you can now login to the device locally using your Unifi credentials + 2FA, with a nice false sense of security.
So in other words, if your account creds and TOTP secret got leaked, and you did not change them, anyone who gains access to your local device's login page can still own you.
This remains true even if you disable Remote Access. (And in fact, as of recent firmwares, the ONLY way to disable remote access is through the Unifi cloud panel. You can't even do it locally.) It will still call out to Unifi's cloud service and authenticate you. There's no way to turn off this SSO account. The best you can do is block all access to Unifi at the firewall, and download all firmware updates manually. And good luck with that if you are using a Unifi device as your gateway.
Like, the screen you get if you hit up your UDM's public IP, or go to the network controller (private IP) and open settings -> controller settings -> dream machine
I don't see how to remove the remote login from the Cloud Key, I tried creating a new Super-admin, but there's seemingly no way to remove the user.
I guess I'll just have to factory reset them and start over with all new credentials.
But I also hope Ubiquiti performs a full audit (ideally by an expert 3rd party) to ensure the integrity of their build systems and reassure customers that the worst hasn't happened. Because it's been radio silence on that part, and that they haven't even announced they're planning on it makes it seem like a blindspot.
This is quite the leap in logic.
The whistleblower's claim:
> They were able to get cryptographic secrets for single sign-on cookies and remote access, full source code control contents, and signing keys exfiltration
And what Ubiquiti has further stated:
> we have well-developed evidence that the perpetrator is an individual with intricate knowledge of our cloud infrastructure.
So someone with intricate knowledge of their infrastructure and access to their source code + signing keys, is doing what exactly with those VMs?
I'm not trying to 'prove' that the firmware is compromised. We can't do that with the information we have. I'm just highlighting strong evidence that the possibility needs to be thoroughly investigated.
I believe this controller version is December 2020 and predates the January issue disclosure date(though possibly not the incident).
I have suricata taps on my net that are clean. Aps and controller are predictably trying to get to their telemetry call homes but nothing more.
That said my Ubnt controller and aps have almost no internet access and haven’t for a long time. I only enable it when performing upgrades and disable it.
I’ll pull the firewall logs and see specifically the IPs they are attempting to hit.
MikroTik isn’t even in the same league as UniFi. Their interfaces and docs are terrible. The only reason I ever considered buying something from them is because they had a good price for a 10gb switch. If you want a dumb device you can’t beat their price.
Check out a few of the reviews on Amazon. Seed and Don’s reviews are very critical.
Is there anything in this space that is good hardware with a decent UI? When I looked the options were consumer crap or very expensive enterprise hardware with no UI.
I'm not sure if the UI is 'consumer friendly' but I didn't need the commandline and I had never worked with any HP networking gear prior to that, and had no difficulties.
The problem is there are too many features packed into a an interface that could be mistaken for something designed 20 years ago.
For some people that is not an issue, but it is a barrier to them competing with the mainstream devices.
Oh yeah, UniFi APs seem to need rebooting periodically to keep working well. My old OpenWRT gear was rock solid with a year of uptime.
I’m currently migrating away. I’m sick of it.
Running that stuff on a UPS is super beneficial regardless.
Decades ago, I was a network engineer working in production data centers. I’m used to setting up switches manually. But I have no professional experience with setting up meshed WiFi networks. I like that the Ubiquiti APs “just work” for the most part.
I’ve started to notice weird AP/client failures with my ubiquiti gear and debugging is a nightmare. It seems that ubiquiti has hidden a lot of useful information in an overzealous effort to streamline UX.
I've never seen any Mikrotik product available to ship, it's always on back order. I've even had resellers call me up and offer alternatives because my order was on back order.
I observed the UI was hiding the ability to apply updates, SSH was enabled, but it didn't allow me to connect and the name of it was changed to what looked like the MAC address.
No permissions were changed, the UDM was sitting unattended and the other administrative user did not update their password prior to this happening.
I am very paranoid and ended up doing a full reset as I was only able to partially administrate it.
Has anyone else had a similar experience?
P.S. The full reset did seem to resolve it
I was considering a the Odroid-h2+ from hardkernel. It has an Intel J4115 (4 cores, 10 watt TDP), 2 Sodimms (max 32GB ram), and two 2.5Gbit ethernet ports. Looks like it would make a decent router. There's an optional board to add 4 more 2.5Gbit ethernet ports.
Wasn't sure on the OS, OpenWRT is definitely in the running. Or maybe VyOS (the open fork of vyatta), which is relatively close to what Ubiquiti ships.
https://www.amazon.com/dp/B01H2QJTM4/ref=cm_sw_r_cp_apa_glc_...
And put Opnsense on it.
Guaranteed security updates, clean GUI interface, open source. Set it up once then forget it.
If you want an AP there are many options on Amazon. For maximum security prevent the AP from being able to access the internet.
This 'about' page doesn't say very much... https://protectli.com/about/
And they won't be "pwned", unless you're running closed source non-updatable software on them.
I bought an H2+ sometime last year to use as a router but struggled to get acceptable networking performance, so moved onto something else.
They look pretty good, managing 11Gbit/sec or so @ 40% CPU utilization with iperf3.
Did you upgrade the bios as mentioned on that page "Have to flash GLK-ESF BIOS to use H2 Net Card properly"
My use case is for a home, ideally with a NAS and 2 desktops on 2.5 Gbe with only a single client at a time using more than a gigabit for large file transfers.
Did you find anything else with 4-6 2.5 Gbe ports?
You need the drivers though. Looks like rtl8125b support landed in Linux 5.9, but as far as I can tell VyOS and OpenWRT are still on 4.x, so you'll likely be in custom land - either upgrading to 5.9, backporting the driver, or using the realtek-provided driver. Looks like people are working on it recently for openwrt: https://forum.openwrt.org/t/setup-an-odroid-h2-with-openwrt/...
I'm only using gigabit at home, so didn't need 2.5g.
I do often move 20-40GB files around, so the bandwidth would actually be useful. Apparently ubuntu-20.04 has the right drives, but thanks for the info, seems like I might need to upgrade to a semi-current kernel to get the interfaces working.
I often tell people when they ask for company/product recommendations that I can only really recommend somebody after they've screwed up.
Any company can deliver product without hassle, but the ones you want to trust are the ones who handle mistakes properly. In this case, I just lost all my confidence in Ubiquiti because they've done the worst thing when it comes to maintaining my trust, lie and deceive.
But it seems that's been getting more and more difficult. Yeesh.
That said, my UAP-AC-PRO at home has been on the fritz lately and I'm a bit lost on what I should upgrade to.
This news comes on top of a lot of issues I've had with them lately, mostly around firmware updates and broken promises.
Am I missing something with all these comments talking about “needing” a cloud account? I have two of their APs and a USG, and I don’t do anything in the cloud. I have a Cloud Key on my network, but that was just for convenience. I could’ve run their management software locally.
I've grown into many of the ER features over the years. Also fun fact - the storage is literally a USB flash drive plugged into a standard USB-A port under the cover! (which is totally upgradable). I will forever be impressed with this product.
That's quite interesting. You can imagine a disgruntled, former employee that didn't lose Lastpass or AWS access... or someone else very close to the company?
It’s close to importance to vaccines. Did you know vaccine safety regulators see the data from a trial before the company management? They operate independently because trust and transparency is so absolutely crucial for both the product and public health.
We rely on routers quite a bit in society to secure our identities, personal transactions, and critical infrastructure: we should be applying the same mindset that we do to vaccine safety to router security!
Are there any decent consumer alternatives if I'm done with Ubiquiti?
If you find something better, please post.
3 APs on a lower setting would give you much better performance.
1 AP is a hard requirement, sadly. I have 6 locations in my house I can plug in my cable modem and for now there shall be no hole drilling and Ethernet cable running. My current setup works, but I'm surprised I cannot just centrally locate 1 Wifi AP. We did that when I was younger in a far larger house. But I guess back then wifi was far slower and more robust?
Also, 5.8GHz does not penetrate walls as well as the older 2.4GHz standards.
1. In the past you were probably the only person with wifi in a mile. Now you have countless units attempting to talk in any given place on a very limited spectrum.
2. 5ghz is fast but walls and objects kill your speed quick.
3. Turning down your current 2.4ghz to wireless B speeds will increase your reliability in environments with few neighbours. Imagine a 10mbit network hub. Worked ok back in the day. Now try to increase its speed to 1gbit and add lots of uses. You wont get much done, everything gets corrupted and you retransmit a lot.
You found this worked well?
In my home, I'm using a wired backhaul, which is the prefered option if possible. My only critisism is that the Orbi Wifi 6 system is quite expensive and there probably exist cheaper options to achieve the same.
It will dramatically help performance compared to mesh that have to pause tx to talk on the back haul.
In the end I went with 3 Unifi APs + a controller. You can use wired or wireless for backhaul. The main thing is that the APs have to be able to communicate with each other.
There isn't a great way to test out the wireless backhaul without buying the equipment. I suppose you could use a scanner to test the RX signal strength with your AP in varying places. If the 2.4/5ghz signal is strong in the (AP, scanner location) pairs then you know the APs might be able to talk to each other.
If you're looking for a less work setup, the Amplifi line, or Eeros might work.
They've now shifted to targeting consumers and their new stuff doesn't even have the basic features of other consumer hardware. Want DDNS? Something found in every other consumer router I've used. Nope. Want an admin console to configure or update without your smartphone (or when your wifi is down again)? Nope. Want to set a static IP for a device? Ok, but we restart the router each time you set one. Want your network to be bigger than /24? Nope. You can checkout their forums and see how long they've been aware consumers want these things and they've delivered on so little.
I like ubiquiti because it's pretty flexible, powerful, I can keep config files in git, IPv6 support is good (I have a /64 per router port), and I can separate router and APs, and handle multiple APs well (without mesh/forwarding).
I have been a Ubiquiti customer and proponent for a few reasons:
1. I have two houses, and the cloud feature lets me easily connect to and manage my devices at the other house when I'm not there. Not a frequent use case, but it does happen, and this just makes it easier. Sure I could setup some other remote access options, but this just works out of the box. I also have their security cams and I can easily monitor those remotely, though the video is stored locally at that location.
2. They occupy a middle between consumer gear and enterprise gear. Giving you flexibility and features that you can't really find in consumer gear.They have a fairly wide and frequently updated product line. One house has ethernet jacks in every room, and for a couple of rooms, I replaced those jacks with Unifi In-Wall HD access points. These are great because they are out of the way, you don't notice them, and they still provide wired ethernet ports. So my office has strong Wifi but I also connect my desktop PC to Gig ethernet via the in-wall AP.
At the other house I use the Flex HD access points connected to MOCA adaptors (the one part of my network not made by Ubiquiti) and just tuck them out of the way.
3. They have a fairly well integrated set of products. I have routers, gateways, switches, and APs, all discrete devices, but managed from the same interface. Managed on the web or on my phone. If I add a new AP to a location, I can just adopt it into the network using the app on my phone, the firmware gets updated and it becomes fully managed, taking on the default settings for that site.
All that said, I still really like their product lines, but this episode made me lose a lot of trust.
I think part of the problem is that effectively everybody has flaws, so you're forced to choose who has the least-broken approach for your needs...and you can't do that without a lot of research.