Hacker News Reader

Comment by LinuxBender | Hacker News Reader

Are these physical servers or VM's? I ask because some VM types can be frozen/shapshot/cloned or live replicated including memory contents as is done in some VPS providers for lawful requests. From the bare metal host anything can be accessed in a VM or container. Do they have a diagram of their physical setup?

[Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?

View on HN

Mullvad has been pushing system transparency a lot (which is a security system specifically designed for bare-metal), so it's fairly safe to say that it's based on bare-metal.

https://www.system-transparency.org

Maybe it should be safe to say? but do they actually have a diagram of their physical and logical setup that shows they use bare metal only and not VM's or containers? Is that documented in audited security controls?

Such a weird discussion because we're talking about a company that is going from the very fundamentals of operating system design and the boot process of bare metal machines in order to get more security.

It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security.

Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question.

They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...

It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.

> Containers, btw, cannot be snapshotted.

lxc-snapshot(1), podman-container-checkpoint(1), and docker-checkpoint-create(1) all beg to differ.

Some (maybe all) of these are backed by CRIU: https://criu.org/Main_Page

Docker at least is

I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

Containers, btw, cannot be snapshotted.

I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted.

They've also discussed booting UEFI directly to VPN nodes:

That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this.

It's completely weird to argue that they might introduce a layer of insecurity here.

Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.

>They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit.

>On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant.

>After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.

https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...

I remember that and that case was cool. I will leave my questions in place though because I recall Apple doing the same thing when a terrorist phone was locked and they claimed to not be able to access with a big public show/fight it but turns out they could and so could the FBI. My theory is that they did not want the public to lose confidence in their phones.

Apple would've had to create a specific OS update with a security hole the FBI could've exploited and distributed that.

Courts couldn't force them to do that -> FBI went another way. This was in the iPhone 7 era IIRC.

Currently their stuff is locked down even tighter and Apple has even less ability to hack anyone's phone. Barring a full-on backdoored software update targeted to a specific person - which they refused to do once already.

Yeah, that Apple case was really interesting. I always guessed that Apple could simply push a software update (possibly to everyone) that happens to open a backdoor for a specific phone. I wonder if that's what they did.

What happened in that case is the FBI went to a different vendor, and they broke into the iPhone either with a zero day they had developed or more likely, they just cloned the phone hardware thousands of times until they could guess the password.

Yep, Apple had nothing to do with it. They do hand over iCloud data (which now can be E2E)

Maybe - depends on how secure enclave is built. It may have hardware defined limitations on # of tries for the passkey and no way to circumvent that in firmware even.

> I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.

"Here is a subpoena compelling you to disclose the data you have on XYZ."

"Sure. Here is the data we have on XYZ." hands over blank page

It's not sidestepping the request. They literally do not have the data, because they don't retain it. And unless there is a specific law mandating that they retain the data, law enforcement have no grounds for punitive action.

> Containers, btw, cannot be snapshotted

Why not?

criu has been used for live migration for a while.

Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.

I think it's a very rational question to ask. Is Mullvad using only bare-metal infrastructure, or is there any logical abstractions added to the servers?

Companies make statements all the time which can be twisted to their benefit, but read exactly like what a customer is after. I'm not stating Mullvad does or doesn't do this, btw.

Even if they did... if mullvad was a honeypot, there is nothing that would stop them from posting a fake diagram.

Physical server can also be frozen (literally) and then data extracted from RAM.

It can but to my knowledge law enforcement in the US anyway have only done this once for a national security incident in the real world outside of a lab. If you have any links to this becoming common practice outside of a lab demo or security presentation on their own hardware I would be interested to save in my own documentation.

They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went.

[Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now.

    init_on_alloc=1 init_on_free=1

There exist special UPSes that you can attach to a power line of a server, unplug and unmount it, and keep the server powered all the time, without the server being able to detect that.

If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.

Just like Seinfeld and saving the Frogger high score.

Whoa! Any keyword suggestions to find those UPSes?

https://wiebetech.com/products/hotplug-field-kit/

I'm having trouble remembering what they're called, but I do remember reading about these. I saw a demonstration where they slightly pulled at the desktop power plug in order to hook a thin device around the plug connectors to provide power, and then they could transport the desktop around.

For laptops, law enforcement use "mouse jigglers" to prevent screen locks and laptops going to sleep, which I guess is similar. Random article: https://www.electronicproducts.com/what-is-a-mouse-jiggler-a...

It becomes way more straightforward if it's a server with redundant power supplies.

Actually not because a server with multiple PSUs definitely knows when one loses power. It’s made to raise alerts about a power rail failing. So this makes it harder because you need to keep power on both to not raise any alarm

That is why modern servers have the option to encrypt the DRAM to prevent this attack.

Moreover, there is also the option to have distinct randomly-generated keys for each virtual machine, to make useless the speculative attacks that succeed to peek at the memory of another VM.

AMD has offered these options for years and the future Intel CPUs will also have them.

good ole can of air compressor flipped upside down and quick hands trick

Are there any known examples of this actually being done by law enforcement or government and working?

There are plenty of examples of it working (e.g. researchers from F-Secure presented at SEC-T conference in 2018 with a cold-boot against Bitlocker), but I haven't seen a LEA or government come out and say "we got access to X via a cold-boot attack".

Although, obviously, it was the same story with dragnet spying before Snowden. Sure it was technically possible in a lab environment and law enforcement had incentive to do it, but nobody could turn up any actual evidence.

As RAM densities/speed increase, I wonder if this is still viable. DDR5 had to mandate some baseline ECC given the intrinsic error rate.

The more I think about this VM's are not required. There are tools that can extract data streams based on specific parameters on demand using eBPF. Perhaps tools like Sysdig/Falco, etc... It might also be useful to know if they custom compile kernels to disable eBPF and if they do any additional specific kernel hardening to strip out or disable on demand debugging capabilities.

One can fix that by deploying software with the VM pause functionality removed and / or remove disks the bare-metal servers and only PXE-boot them. That way there is nothing to pause.

If you are asking this, you probably need to self host your VPN

I do this but I do not use them for privacy. I use Tinc [1] to route around internet outages as it can do user-space dynamic mesh routing. Since they are all nodes I pay for they would not hide who I am. While I do control the logging of Tinc I do not control external logging of the server and VPS providers.

For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi.

[1] - https://www.tinc-vpn.org/

Reply on news.ycombinator.com