Moving our Encrypted DNS servers to run in RAM
mullvad.net
mullvad.net
In addition they also support optional content blocking[1] via blocklists, just set the desired TLS/HTTPS DNS server.
Little Snitch / Lulu die?
Do you have like a ticket for more explanation?
> If you have more than one profile added then macOS seems to use only the last one that you installed. If you remove a profile then it appears that it will not use any of the other profiles.
This isn't unique to Little Snitch/Lulu and Mullvad though, you would run into the same problem with any application that creates a network filter by using a profile, see [1] for example. Since the full blown Mullvad VPN app does not use system profiles, it does work in tandem with Little Snitch/Lulu.
[0] https://mullvad.net/en/help/dns-over-https-and-dns-over-tls#...
[1] https://git.codeproxy.net/paulmillr/encrypted-dns/issues/13
I’ve never been able to get CrowdStrike Falcon and Little Snitch to work on the same machine because of this. It would be nice if you could have n filters (and a limit on n isn’t unreasonable).
Not like in iOS, DNS profile work very well.
As I see it, the model is that I need to contact a site, and use DNS to get the IP - but either my ISP or my VPN provider can see I’m visiting that IP whether I looked it up just now, or knew it already.
So unless the model is I’m using a VPN, but am leaking via someone else’s logging DNS, I don’t really get it.
> Queries could be directed to a resolver that performs DNS hijacking. For example, in the UK, Virgin Media and BT return a fake response for domains that do not exist, redirecting users to a search page. This redirection is possible because the computer/phone blindly trusts the DNS resolver that was advertised using DHCP by the ISP-provided gateway router.
[1] https://www.cloudflare.com/learning/ssl/what-is-encrypted-sn...
Sure, you might have a SOCKS proxy configured for the HTTP request, but depending on how the program makes the DNS query, it might bypass that proxy: https://support.torproject.org/misc/check-socks-dns-leaks/
Encrypted DNS isn't a perfect solution, because the request still won't go through the proxy, but at least it won't leak the info to your ISP (assuming you have something else configured as your DNS provider.)
Obviously, this isn't limited to just Tor.
A vpn reduces speed and battery life
One scenario comes to mind: multiple services under the same IP. “cutecats.com” and “hotintercourse.net” might both resolve to the same IP, but a reverse proxy resolves requests differently depending on the host. Then, knowing the IP but not the host, as your ISP does, becomes a meaningful difference.
[Edit] - Are there any Mullvad architects here that can help us avoid going down theoretical hypothetical rabbit holes and turtle stacks?
It's very antithetical to that goal to add more layers of abstraction that could be buggy or reduce security. We're talking about motivated and intelligent people purposefully trading off convenience for security.
Containers, btw, cannot be snapshotted. So that's a weird thing to put into your question.
They've also discussed booting UEFI directly to VPN nodes: https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
It's completely weird to argue that they might introduce a layer of insecurity here. Directly goes against everything else they're working on.
Why not?
Getting a consistent snapshot that you can restore reliably is a hard problem but probably not as big deal for forensic analysis.
I am extra skeptical of a company that pushes this. They and I know they can't side step lawful requests which raises the spidey senses even further so I believe it is a valid question.
Containers, btw, cannot be snapshotted.
I did not say they could be. Their memory contents however can be accessed, even if the host memory is encrypted.
They've also discussed booting UEFI directly to VPN nodes:
That in no way precludes having VM's. I have run VM's on PXE Diskless nodes. The boot method is orthogonal to this.
It's completely weird to argue that they might introduce a layer of insecurity here.
Weird maybe? But completely logical and valid question nonetheless. They are leaving 53 characters out of their documentation unless I missed it. My questions could be solved by saying "We do not any form of virtual machines or containers". That is only 53 characters and should fit on their document site.
They have been able to turn down lawful requests previously, which is (at the very least) a positive indicator that may lower your spidey senses a bit.
>On April 18 at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant.
>After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
Courts couldn't force them to do that -> FBI went another way. This was in the iPhone 7 era IIRC.
Currently their stuff is locked down even tighter and Apple has even less ability to hack anyone's phone. Barring a full-on backdoored software update targeted to a specific person - which they refused to do once already.
"Here is a subpoena compelling you to disclose the data you have on XYZ."
"Sure. Here is the data we have on XYZ." hands over blank page
It's not sidestepping the request. They literally do not have the data, because they don't retain it. And unless there is a specific law mandating that they retain the data, law enforcement have no grounds for punitive action.
lxc-snapshot(1), podman-container-checkpoint(1), and docker-checkpoint-create(1) all beg to differ.
Companies make statements all the time which can be twisted to their benefit, but read exactly like what a customer is after. I'm not stating Mullvad does or doesn't do this, btw.
They can also buy cases that have anti-tampering controls that force a poweroff if the case is opened but that is getting into more costly edge cases. The hardware exists but is not datacenter operations friendly. It is also possible to force a wipe of ram on a clean shutdown but one must assume this won't be. As one example the feds took Mega's servers from Equinix after yanking out the cables and out the door they went.
[Edit] I should add for completeness sake there are kernel boot options to clear memory space before allocation and after de-allocation with a performance hit. Both are enabled by default on modern kernels now.
init_on_alloc=1 init_on_free=1If you are really serious about security, you need to have a watchdog on network link downtimes because, no matter what, you can't disrupt a fiber optic network link without the server noticing.
Moreover, there is also the option to have distinct randomly-generated keys for each virtual machine, to make useless the speculative attacks that succeed to peek at the memory of another VM.
AMD has offered these options for years and the future Intel CPUs will also have them.
For privacy I might theoretically use Tor but most sites block or grief people on that network. Tor exit nodes share some risks that of public WiFi.
Can we reliably "forget" the contents of what's in RAM by powering-off and letting some encryption key somewhere fade from a smaller, more volatile piece of memory? I'm curious what has been done to mitigate cold-boot attacks in RAM. What has changed in hardware? Are the keys to encrypt contents of RAM kept in the TPM or something?
https://www.intel.com/content/www/us/en/developer/articles/n...
Looks like AMD has an equivalent:
https://www.tomshardware.com/news/intel-mktme-amd-memory-enc...
I wish there were a sort of checklist that prints out in dmesg saying "Your RAM is encrypted: (check) Your machine has these mitigations against cold-boot attacks: ...some list here... (check, check, check)"
In some contexts, I love how far we've come with secure boot, attestation, speculative execution mitigations, etc. but it's very difficult for the average Linux user to understand what they should expect for hardware security, and then work toward enabling all of it.
I love what Gnome did in the last year in Settings -> Device Security:
https://www.phoronix.com/news/Ubuntu-No-GNOME-Device-Securit...
There's a detailed list here of what it's looking for:
https://forums.debian.net/viewtopic.php?t=152705
Not a fan of all the bound-to-Intel technologies. I wish the concept of what's being secured were identified, then made available in dmesg or through some /sys or /proc interface.
Intel is only now catching up.
[ 0.338594] Memory Encryption Features active: AMD SMEEncrypting what's in RAM would possibly add some additional protection but realistically if things are done right what's left in RAM will probably be only marginally more interesting than what you can see over the wire; at most I'd expect you could see a tiny amount of active request data. That's just a guess, though.
Does this prevent rootkits?
> but realistically if things are done right what's left in RAM will probably be only marginally more interesting
That's the idea. If the OS is immutable, where do you put them?
Bootkits require further mitigation to prevent, but Mullvad has also put effort into that, too, with their secure coreboot bootchain.
Of course as an end-user you still can't really fully validate a machine over the network has not been tampered with, so you're obviously ultimately still trusting Mullvad as you'd expect.
> See: https://en.wikipedia.org/wiki/Cold_boot_attack
Yes, that's what I'm referring to. I don't expect that there would be a significant amount of requests in RAM, probably just remnants of mainly what was active (and some de-allocated but not cleared remnants?) when the machine was reset.
They would be in the PXE image. There was one time I flubbed on this actually. I had to change export options on our PXE servers to fix something and I forgot to change it back adding my poor excuse of multitasking and being lazy. This was in a Dev environment we also used PXE in production believe it or not. A developer sitting across from me said he changed something in the image by mistake and I said, "That's not possi... oh crap." It was cool of them to call it out right away. Attackers probably won't be so kind.
https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...
secure boot, as painful it is to get right, doesn't help when the vendor runs its own things with elevated privileges, like they are known to do: https://news.ycombinator.com/item?id=34085635
Doubly so, when the said things are not only closed source but vulnerable themselves: https://archive.is/6ClWm
Does coreboot help in mitigating vuln in privilged vendor blobs?
Mitigating the risk of Intel ME is obviously something that is complicated. The "state of the art" so-to-speak is really just rendering it non-functional. This isn't strictly related to Coreboot; that said, Coreboot has a page about ME:
https://www.coreboot.org/Intel_Management_Engine
And in general, when you're running Coreboot, you're running significantly less privileged blackbox blobs (possibly none in some rare cases), and in general, the surface area of Coreboot is going to be dramatically tinier than any typical UEFI system firmware. So it's absolutely worth it in that regard.
There's never going to be a "perfect" option here with how complex things have gotten, but I think that you can at least say that Mullvad has taken almost any reasonable step, and perhaps a few steps that are beyond what many would consider reasonable, towards developing effective mitigation of the risks of rootkits, bootkits and malicious/unwanted vendor code running on their "diskless" architecture.
Of course, you still have to trust Mullvad. Trust is definitely a meaty human-y experience, and the cold rational part of our brain craves some kind of technically-bulletproof way to prove that it's truly safe, secure, and private, but that's just not possible. Best you can do is layer enough assurances on top of each-other in a way that at least some layers are completely independent of others for maintaining your operational security. Far easier said than done, and obviously how rigorously one approaches this will vary on paranoia.
At some point you have to pick your middle-ground I suppose.
Wait why would this be a bad thing?
No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed).
There's even an EFI reboot flag that could fix this and force-clears the RAM on next boot, but it's usually deactivated.
[1] https://en.wikipedia.org/wiki/Cold_boot_attack (works still on Windows, thanks to BitLocker)
[2] https://www.usenix.org/legacy/event/sec08/tech/full_papers/h...
Bitlocker does not store the keys in the CPU, otherwise a coldboot attack while literally transplanting the physical memory rows to a different machine wouldn't work. And it does.
In case you are confusing CPU with a TPM chip, which can optionally store the master volume key but only for activated full disk encryption:
With TPM you can easily bypass that, too, with tools that can read the i2c bus. See here [1]
Any non-ridiculous RAM encryption will keep the key in the CPU. And since the key only needs to last for one boot, it never needs to leave the chip. If there's hardware in the memory controller doing the encryption it can be designed so that key export is impossible.
Game consoles have been using RAM encryption for over a decade. It's one of the reasons consoles are much harder to attack nowadays.
First: Only Ryzen PRO or EPYC models support it, which kind of kicks out all mobile or laptop systems already. Then, only Zen3 CPUs work, because previous generations have a boot freeze bug, which wasn't fixed and upstream linux 5.15 as a result disabled the mem encrypt flag by default.
Second: Before you switch topic to SEV, that's only supported for EPYC models, see here [2]
Regarding attacks: At least AMD had an injection attack problem where SEV in EPYC 7xxx and 3xxx processors was confirmed to be affected without AMD confirming the vulnerability (yet...). It was a master thesis iirc from a guy in luebeck.
There are also known sidechannel attacks which void RAM encryption in practice, like Hertzbleed which used frequency scaling to decrypt ECDSA and PIKE SIDH (which is meanwhile known to be unsecure, at least for PIKE). [3]
Google also did an audit on Intel's TDX where they found bugs in loop boundaries, off by one errors and similar feasible attack methods (which haven't been published as a PoC yet, so I grant you that). [4]
So I would still argue that with these very narrow set of available processors (Intel Pro 13th generation for TME and EPYC 7xxx that have both SME and SEV) is highly limited in its availability and also not available for laptop hardware due to them being server CPUs.
Additionally there's been a lot of attack surfaces that have been proven to have access to SME or SEV stored keys in the CPU and there have been other sidechannel attacks which conceptionally are very unlikely to be fixed anytime soon.
So I would still argue that memory encryption in practice is unreliable.
[1] https://lists.freedesktop.org/archives/amd-gfx/2021-October/...
[2] https://github.com/AMDESE/AMDSEV/issues/1
[3] https://www.hertzbleed.com/
[4] https://services.google.com/fh/files/misc/intel_tdx_-_full_r...
Edit: found the AMD injection attack thesis:
[01] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
Another attack on SEV, which was confirmed by others since the USENIX conference. Both of the techniques rely heavily on pattern matching to find the decryption oracles though, and around 16 bytes for their OpenSSH demonstrations.
[02] https://www.its.uni-luebeck.de/fileadmin/files/theses/MA_Luc...
I'm gonna ignore the game consoles statement because that's a specialized set of CPUs which will never land in any laptop anytime soon. Most of them are AMD, so I'd refer to the same articles anyways.
These are a perfectly good way to back up the first half of your original statement, "No. At least not reliably"
But "due to how the keys are stored in RAM" is not true. And that's the part that got objected to.
You're not being nitpicked on a technicality here. You mixed up the techs you were talking about in a way that makes a significant difference. Bringing in all these references might be useful to people, but you could have just said "oh, I meant disk encryption by that part" instead of getting sassy and doubling down on saying incorrect things with "Did you read any of my referenced links? You cannot state this when the paper and well known attack methods disprove your statement."
If you gonna rip apart my comment into parts without context, at least do it with all parts.
Note that I didn't start the discussion about where the key is stored, someone replied to my initial comment - in the context of BitLocker - saying that the CPU stores the key, which is incorrect.
My initial point was not about how secure enclaves work in practice. Though I argued that most implemented follow-up technologies don't work reliably due to how the OS always interfaces with unpatchable firmware bugs.
While I agree this is the only way to fix the coldboot attack problem I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
If you don't agree: show papers with counter measures, and how to prevent the exploits and bypasses.
You did mention bitlocker. But it was unrelated to RAM encryption, and you acted like it was related.
Everything you said about bitlocker was true, but the way you brought up bitlocker was itself the problem.
> someone replied to my initial comment - in the context of BitLocker
No they did not. Your answer did not make a new separate context for just bitlocker. You muddled bitlocker and RAM encryption. They replied in the original context, because you said that RAM encryption stores the key in RAM during standby.
Whether you intended to say that or not, it's what you said. Go back and read the original comments. You directly said that line about RAM encryption, not just disk encryption.
"> > As a naive outsider: Does RAM in 2023 encrypt all its contents?"
"> No. At least not reliably, due to how the keys are stored in RAM as well (e.g. when in standby mode, or when a laptop lid is just closed)."
> I don't see the technology as reliable as of today, and I don't understand why you think I am wrong with that statement?
I never said you were wrong about that.
Granted, I should have formulated it probably differently and specifically differed between secure enclaves, memory encryption and full disk encryption problems to be more clear about it.
https://www.amd.com/system/files/documents/amd-memory-guard-...
https://www.amd.com/content/dam/amd/en/documents/epyc-busine...
That way Mullvad can only see the IPs I visit, Cloudflare only see DNS requests coming from the VPN IP, and my ISP sees nothing.
Mullvad's DoH DNS seems nice, but it provides potentially less privacy than the above, so I won't be using it.
I hate to break it to you, but if they wanted to look at the SNI on all your TLS connections, they could; ECH is not really a thing yet.
My point stands though about not putting all eggs in the one basket and using a different DNS provider than that of the VPN provider.
But in terms of clawing raw performance decimals, I applaud the effort. This would be a fun redis project.
The majority of systems I've worked on used a message queue of some kind for logging rather than a file on disk.
I suppose it could be useful to expedite downtime in the case that they are subpoenaed, but I doubt claiming to be diskless would prevent any subpoena.
> I doubt claiming to be diskless would prevent any subpoena
It does in Sweden. They've repeatedly squashed subpoenas
> long as it is connected to a network you cannot prove a system to be log-free.
Nothing is perfect. If you read their audits, it's clear admins could get at your traffic if they wanted
https://mullvad.net/en/blog/2022/6/22/vpn-server-audit-found...
It has already.
This quote is in reference to ProtonMail in 2019, a fairly widely reported on case at the time.
This would be a point in favor of Mullvad, as Mullvad has demonstrably squashed a subpoena in the past, thanks to their data-minimization policies (no PII for signup, no PII options for payment, diskless infrastructure, etc.).
Note that this is more related to privacy, and not as much security. If the parent comment was specifically talking about security, I would recommend reviewing the software and infrastructure audits made available by either company. I'm not aware of any major fault with either, from a security perspective.
You can find many articles about it (search "Protonmail subpoena 2019"), but an example article is here: https://www.privacyaffairs.com/protonmail-surrenders-user-lo...
ProtonMail's own blog post about it is here: https://proton.me/blog/climate-activist-arrest
That being said, Swiss law is very restrictive, and there are a lot of hurdles that one needs to jump through to get a court order. And even with a court order (and has been proved multiple times in court), there is no way to break Proton Mail's encryption. Privacy is not the same as anonymity, and due to the way the internet works, if anonymity is what you are going after, you have to exercise proper infosec and take preventive measures, such as using Tor or VPN.
Under Swiss law, the treatment of VPNs is different, and VPNs can indeed be no-logs, like Proton VPN is. No-logs VPN is also possible in other countries as well. What makes Switzerland different, and possibly unique, is that within the current Swiss legal framework, Proton VPN also does not have forced logging obligations. You can find out more here: https://protonvpn.com/blog/transparency-report/.
Everything runs on RAM and will disappear the second a server is shutdown, they really don't store anything.
And you can buy a Mullvad account by mailing cash to them or with crypto. No need to create an account with an email address or anything.
>We recently announced the completion of our migration to remove all traces of disks in use on our VPN infrastructure.
Is this more clear?
> On April 18 [2023] at least six police officers from the National Operations Department (NOA) of the Swedish Police visited the Mullvad VPN office in Gothenburg with a search warrant. They intended to seize computers with customer data.
> In line with our policies such customer data did not exist. We argued they had no reason to expect to find what they were looking for and any seizures would therefore be illegal under Swedish law. After demonstrating that this is indeed how our service works and them consulting the prosecutor they left without taking anything and without any customer information.
https://mullvad.net/en/blog/2023/4/20/mullvad-vpn-was-subjec...
Nonsense. The linked post shows that they didn't have the sought information. If the information was present in RAM, they'd just read it from RAM.
>If the computer is powered off, moved or confiscated, there is no data to retrieve.
>We get the operational benefits of having fewer breakable parts. Disks are among the components that break often. Therefore, switching away from them makes our infrastructure more reliable.
>The operational tasks of setting up and upgrading package versions on servers become faster and easier.
>Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
and here:https://git.glasklar.is/system-transparency/core/system-tran...
The second link talks about network boot mode and signature validation.
You can connect with wireguard and not run any mullvad software on the client side.
Though that's generally the case with VPN providers, whether openvpn, wireguard, etc.
I guess they could introduce some vulnerability on the server side if they rolled their own wireguard implementation or patched it. Though it seems unlikely to me.
https://mullvad.net/en/help/easy-wireguard-mullvad-setup-lin...
Where exactly do services run from if not RAM? The post is very plain with no explanation of what "running from RAM" means. When you execute any binary it gets loaded into RAM as per standard operating system procedures...?