Hacker News Reader

Comment by beckler | Hacker News Reader

We tried to make an internal IoT device using Ubuntu Core and snaps because the capabilities of it were very promising. We started a PoC and about halfway through we hit a major roadblock. Our enterprise network does certificate substitution, and Ubuntu Core absolutely does not allow you to install your own certificates globally, so our devices would never receive updates. We tried EVERY hack we could think up, short of making our own core snap. We talked to Canonical about it, and they didn't seem interested in our fixing our complaints without a massive amount of money, so our PoC died, and we dropped Ubuntu entirely because of it.

View on HN

Sounds like a solid decision from Canonical tbh. Certificate substitution is terrible.

MITM should be completely illegal.

Why does this even need to be stated?

Seriously, you can think of no reasons?

Many competent companies MITM employee traffic to scan for malware, leaking confidential data, etc.

Grandparent comment by beckler says they were trying to make some IoT product. That will be deployed in situations where that happens; if your customer has a MITM set up, you just nod your head and sell them something that works in that setup. You can't say, "MITM should be illegal, please buy my non-auto-updating solution anyway and stop it with your MITM."

Good thing beckler found this while eating their own dogfood due to their own network being that way. Imagine that everything worked fine in their environment and then so customers came back with this issue. Then they would be beavering away hacking up their own core snap or whatever.

Should you not be able to MITM your own machines?

Only in dev environments. Not on machines used by employees.

That is different than "completely illegal".

are you european. because in america its perfectly legal to spy on your employees usage of company assets.

Not if other people are using them.

There are different value tradeoffs in different countries. The US says it is okay to spy on employees for no reason at all as long as you use company equipment. The EU says that employees like every other human being have rights and you better have a good reason and do so in a respectful way and be clear about it.

For some situations, it's called for, but it's a huge pain in the ass. I am in a similar situation, and I need to patch every docker image I use. It's terrible to deal with, as an engineer, but the information security team does catch and eliminate a lot of content-based attacks.

I agree its a pain. It also makes things like working with other private certificate authorities (DoD Cert authority, other private certs) a pain. I spent a decent amount of time trying to get certain work/project related sites whitelisted from our MITM proxy because it didn't recognize the certificate chain...

In your own company you're free to do what you want.

I can understand the reason for this. Now that most suppliers treat their devices as 'black boxes' and call home to install updates whenever they want, the security team no longer has visibility nor control over this. So much stuff runs Linux which we don't manage but still has to have full access to our network.

And public repositories have been compromised and spread malware in the past. So yeah I totally understand this, even though as an enterprise Admin it's a total PITA to manage the root CAs.

> In your own company you're free to do what you want.

No, it's corporate MITM specifically which should be illegal.

Why? Companies need to control what goes in and what goes out.

Yeah, I want traders to be able to communicate and trade freely away from the prying eyes of their... employer?

Not gonna disagree at all, but I don't see any widespread adoption from enterprises because of it. It's disappointing because Ubuntu Core is actually quite secure, and we were really impressed with it... we just couldn't use it.

For a consumer, sure. In a business setting, it seems irresponsible to be to allow every random server to have an un-inspectable VPN.

Yep I couldn't imagine our Fortune 500 company ever allowing access without it.

It seems irresponsible to inject devices into your network that that indiscriminately MITM all traffic and can easily be configured to log passwords and auth cookies, no matter what setting you're in.

It's basically the TSA of corporate networks. They need to inspect traffic because they can't control what devices show up in their environments and what malware might ride along side legitimate traffic.

Actually most of these corporations have plenty of controls on their networks preventing the random plugging in of devices into networks. Most of the time they are using something that involves 802.1X.

Plus which, it allows me to check what black box software is doing. Certificate pinning is great and all, but it also makes it way harder to know what data "huawei mobile services", "google play services", or a random mobile game for that matter, is phoning home about.

I'm not a big fan of these corporate MITM boxes that contain the keys to the TLS traffic of the whole company (which additionally often double as employees' private phones and laptops), but I do like to look at my own device's traffic.

You and I agree. Unfortunately most large corporations, and US Government agencies like to be able to see and inspect network traffic. Mostly to prevent the theft of confidential data. The fact that the MITM proxies hoover up passwords and auth cookies still bothers me quite a bit.

You transmit passwords from servers to internet destinations?

That would be a pretty serious security incident from my POV.

A colleague of mine was also looking at Ubuntu Core for an IoT project recently, but Ubuntu wanted $15k/y to run a private, branded Snap store - erm.. no.

If they really want snaps to succeed, there should be an open source snap store protocol, and 3rd parties should be allowed to run their own stores, just like you can add 3rd part apt repos, for example.

We decided on Photon OS, BTW. It's tiny, and perfect for use as a Docker host.

Wow, it's gone up! When we talked, it was $10k/year, and then there was an additional cost for every update we pushed out. It depended on the speed, size and number of devices receiving the update.

Yes, forgot to mention they also charge for updates! Pricing page[0]

[0] https://ubuntu.com/pricing/devices

Isn't there some old adage about how if you can't afford something you aren't the target audience. At the larger companies I've worked at you didn't even need approval for 15k/y.

This was for a company with just under 300k employees - you need approval from multiple people for everything.

From the marketing, blogs etc, Ubuntu Core does seem to be targeted at everyone, not just people that would drop $15k/y like it was nothing.

It's almost like a trap - it sounds perfect for IoT, so you start wasting your time building a PoC, and then much later you find out about the costs. And as another commenter mentioned, they also charge you for doing updates on top!

At the two larger companies I've worked at (~1,000 and ~50,000), I've been explicitly told that I cannot sign any contracts without getting it approved by the legal department. Furthermore, all software purchases must go through the approval process.

> [..] didn't seem interested in our fixing our complaints without a massive amount of money [..]

Why is asking for money wrong? You want the feature, why shouldn't you pay for it.

Nothing wrong with it, but we couldn't get any funding approved until the PoC was successful, and we couldn't be successful without the feature.

> We tried EVERY hack we could think up

Just to be sure, installing the CA from that MITM box didn't work? Because that should be the generally recommended solution and I can't see why snap would have a hardcoded CA list separate from the system. If that didn't work, it's indeed a bug, but a rather weird one; definitely worth posting to the bug tracker.

The CAs are hard embedded in the core snap. They're pulled from some specific package when built, but snaps themselves are immutable. We attempted to overwrite it in several different ways, but the OS is just simply mounting these folders from the core snap (which is immutable), and then marking those mounted paths them as immutable.

That being said, there are a list of paths you can write to, and they're listed here (for core18): https://github.com/snapcore/core18/blob/master/static/etc/sy...

> The CAs are hard embedded in the core snap.

That is ridiculous. Thanks for the info, yet another reason to stay far away from snap...

What did you end up using?

We ultimately used a custom yocto image.

Reply on news.ycombinator.com