Mullvad: Diskless infrastructure using stboot in beta
mullvad.net
mullvad.net
As an example, let's say that we offered any of our users to at any time during the year show up at our office and inspect our VPN hardware, without warning us beforehand. In that situation, if we wanted to betray your trust and privacy, we would need to put in a lot more effort than if we said "We have secure servers. Trust us on that. No you can't see them.". Does that make sense?
Have you ever thought about doing something like that with some big youtube personalities? Maybe have them hire some pen testers, randomly show up to one of your datacenters, and post recordings of what is done and attacks that could be possible. Since your software is open pen testers could prepare some things to try to attack days in advanced. I'd love to see something like this with Level1Techs or something.
You mean the same people that are being "sponsored" by VPN providers for years? Why would we want to trust youtubers of all people?
It just seems like this goal of using technology to prove they're trustworthy is unlikely to actually work for a VPN company due to the threat models.
At some stage in an R&D project one should shift from exploration to threat model-driven development. Most people, myself included, tend to focus on technical solutions, and argue back and forth how "oh, but it can be broken using X".
System Transparency aims to provide remote auditability assuming (1) the server hardware specification is correct, (2) a correct cryptographic hash of the contents of the SPI flash containing the platform firmware, and (3) a keypair generated on and only accessible to the platform. This is very simplified of course.
An attacker aiming to tap incoming and outgoing network traffic from our servers, who has physical access to the VPN server's Ethernet port, or an upstream router, isn't in the scope of System Transparency to protect against. We need to use other means for that.
To get around this, Mullvad offer very transparent comprehensive multi-hop routing systems [1]; you can bounce your wireguard tunnels around in layered wireguard tunnels (a bit á-la tor) by just choosing a series of ports to tunnel on and to. My understanding is that each one of these adds non-deterministic latency to your connection and probably would help to make such attacks harder at the very least, because from the point of view of an "all seeing" adversary the fact that all of these servers talk to each other all the time makes it very much harder to know where any packet could have gone. Yes, you can see each individual link but the metadata is lost.
I signed up for Mullvad when the UK's Snooper's Charter came into force and the local health inspectors suddenly had the rights to see my DNS record. Since then, I've had it installed on my router and just route everything through a custom wireguard (originally openvpn) tunnel. I've had some issues with my ISP randomly bandwidth limiting traffic on the odd port to 1 MByte/s, but frankly that makes me more inclined to put everything behind an encrypted tunnel. I don't want my ISP to do traffic shaping and I do want them to just leave me alone and let me communicate in peace. I have absolutely nothing to hide, but now have to accept that I partly live in a country where everything is surveilled all the time, and warantless, unaccountable investigation of my (highly personal!) online habits may be happening. I think Mullvad's excellent product, sensible architecture and reasonable price is worth paying. I'm an academic, unlikely to be of interest to three-letter acronyms, and therefore it matches my needs very well.
Here are some additional details for those interested. We intend to make use of TPM for remote attestation of the current boot chain, reproducible builds to provide a strong link from source code to build artifacts, and a transparency log for a historical record of previously used boot chains, artifacts, WireGuard server keys, and related signatures.
As dtx1 mentioned elsewhere in this thread, diskless VPN infrastructure is currently in use by many other VPN providers. That is not a novel feature of course. What is novel is user-auditability of running VPN infrastructure. We were the first VPN provider to state our intention to make our infrastructure user-auditable AND provide a realistic roadmap with the specific technologies needed to do so. See the link above.
I believe the technologies we use in System Transparency will ultimately reshape the VPN provider industry into a highly competitive space focused on maximizing the transparency of VPN infrastructure. Or not, but at least OUR users will be able to audit us. :)
Either way we’re looking forward to the future. The opportunity for improvement is immense.
All source code for all software on our VPN servers must eventually be public, and all build artifacts must be reproducible by 3rd parties.
> I understand that probably you want to keep the database of "active numbers" private, but if I understand you correctly, you want to move to a model where anyone can download your in-memory image, run it in a VM, and audit it independently.
Exactly, but we will also have to measure each artifact in the boot chain into the platform TPM, and allow anyone to issue a challenge to the TPM to get a signed quote of the boot chain measurements.
> I would welcome this. I'm particularly interested in how you maintain access to your bare-metal machines (e.g. do you have ssh / a serial console enabled)
We’ll have to constrain our own ability to access the VPN servers. We cannot be allowed arbitrary root access as that would make the TPM measurements meaningless from an audit perspective. Well, you’d be able to conclude we have root access, so not totally meaningless.
I had one technical support question, and I got an immediate response from an actual person who knew this problem, gave me a simple workaround (toggle between wireless connection and not), and told me that a permanent fix was in the works. Likely that fix rolled out because I haven't seen the issue in months.
The idea of an account that doesn't need a password because there's no critical information saved is such a nice one, too.
Keep up the good work - I recommend you to everyone.
I think I could get behind more of this use!
Great point. I have a hard time comparing the importance of the integrity of our VPN servers with monitoring and logging of network traffic happening upstream of them without a long discussion of the many nuances. Let’s leave it at; they are both very important issues.
> How can you provide transparency of the network?
That’s very hard. Individual AS’s upstream of the VPN server you’re connected to might change their routing at any moment, and suddenly your traffic makes its way through an unexpected AS or jurisdiction, which may change the monitoring situation completely.
Enabling our users to use multiple hops is one mitigation, another is vetting our data center providers. In collaboration with some of our Internet providers in Europe we have deployed protections on layers below IP. Come to think of it, I don’t think we’ve blogged about that. Thanks!
> The law permits the signals intelligence agency, National Defense Radio Establishment, to monitor the content of all cross-border cable-based Internet traffic to combat "external threats" such as terrorism and organized crime.
[0]: https://www.opendemocracy.net/en/can-europe-make-it/didier-b...
[1]: https://en.wikipedia.org/wiki/Internet_in_Sweden#Internet_ce...
I'm not sure what point you're trying to make. If online privacy is as important to them as they say, they could have easily registered their company (or a subsidiary) in a different EU member state.
> They'd have to provide government access if there is a lawsuit that demands it.
IANAL but I am not entirely sure this is true in the EU. Either way, my question was "Why are the servers located in Sweden?" Whether or not this due to the company owning the servers being in Sweden is irrelevant.
Registering the company somewhere else wouldn’t do them any good when they’re living in Sweden, because the legal system isn’t fooled by sleight of hand like that. Likewise, hosting the servers elsewhere from where they’re based. Both would expand the number of entities with the ability to compel them to disclose data, because as long as the company owners live in Sweden, Sweden has that ability.
Unless you’re asking why they didn’t move to another country to start their company, which is surely a larger ask than the “easily” you suggest.
If you believe any different, please say why so. Just "wouldn’t do them any good" is pretty meaningless.
All of their executives and their staff are in Sweden. It doesn’t matter if the company is registered on Mars, the Swedish government can come knock on their doors, because Swedish laws apply to people in Sweden.
The most mundane way to demonstrate this is to imagine they don’t register a company at all. If a bunch of Swedish people get together and start doing business w/o registering a company, it’s clear that Swedish law applies to them. Why would filing some paperwork with a foreign entity grant them immunity from the laws in the country they live and work from?
The second reason is "Swedish laws apply to people in Sweden" seem to make assumptions about what the government can force people to do, or specifically, punish people for not doing. In many cases, authorities just threaten/raid the data-centers so never have to bother take that route.
Lastly, I'm not sure this is true: "Swedish laws apply to people in Sweden" - I'm not sure this applies to Swedes working for foreign corps, there are a whole load of laws that apply to local corps only. In fact, that are laws that apply to Swedish corps even when their staff reside abroad - unless "government can come knock on their doors" is a reference to physical coercion.
I didn’t make any assumptions about what Swedish law can or cannot do. Swedish laws apply to people in Sweden. If Swedish law says that you can’t use Helvetica font on your website, and the punishment is 10 years of hand-tracing a better font on stone tablets, then they’re able to apply that law to a Sweden-based web developer, regardless of whether or not he works for a company that’s registered in Spain.
Likewise, yes, the Swedish government surely has many laws with carve outs for different use cases. Taxes are a great example here: there are laws that apply only to activities of foreign corporations, and laws that apply only to local corporations. But the Swedish government gets to make those laws and determine which apply to whom. Likewise, you are correct that Sweden can make laws that apply to Swedish corporations even when their staff reside abroad. This is because by registering in Sweden, the business has given the Swedish government a measure of control over their activities.
Just because employees reside in Sweden doesn’t mean the company resides in Sweden, legally.
> If Swedish law says ... they’re able to apply that law ...
This is a big assumption, and depends if you mean literally that they can do this, or if they can do so sustainably. Any country can violate international practise, but are unlikely to do so (at least in Europe) because of the consequence on international relations.
A law on Helvetica font would require legal authority. Very often, companies themselves are help liable for the actions of a company - laws that allow the government to punish individuals would have to specifically criminalise the act even for locals acting on behalf of those corps. These kind of laws are much rarer, at least in US/Europe, and not the kind of law we are talking about here which appear to apply to corporations. There is a good reason for this; as soon as any nation officially declares it would punish individuals like this, corps will leave - or at least no longer employ natives into decent positions.
> But the Swedish government gets to make those laws and determine which apply to whom.
Technically, but not really, they have to remain compatible with their international agreements, and their economic ambitions.
But it does matter. In most EU countries limited-liability companies (like the Swedish Aktiebolag) are legal entities that are completely separate from their owners and employees. Your idea of Swedish authorities "knocking" on people's doors (who own a company registered abroad) and "convincing" them to hand over customer data appears to be more along to lines of https://xkcd.com/538/ but in this case (in the particular case of a country like Sweden that has a well-respected legal system) it doesn't seem to be grounded in reality.
For instance, Swedish law likely compells companies to hand over customer data under certain circumstances. But if you're the "just" the owner of that (limited-liability) company, the company's customers are not your customers, so authorities cannot compell you to give them access to those customers' data (because you are a separate legal entity).
> avoid[ing] local laws
(i.e. legally) which is a whole different matter.
It also depends on which country, and to what extent agreements exist between those countries wrt policing their own territories. Those that don't have such agreements, are often also limited in what extent they can do business in the US.
This isn’t true at all, at least as long as we assume that you’re dealing with the courts and not some secret police.
In the US, it is mostly painless to create a company in another state. Fill out some forms online and pay a few hundred. If you don’t have a presence there, you have to pay for a registered agent in the state. That’s about $100/yr. You may also need a mailing address, but you can get a post office box that will accept and scan your mail for another hundred bucks a year. Star and local taxes.
What about the EU?
One example: If you're a cross-border worker you'll likely have to file multiple tax decelerations (one for each country), which can be quite complicated if you don't speak the local language (and even if you do).
Yes, if you have sources of income in multiple countries, you might have to. But if you're (only) working on the other side of the border and this is your only source of income, agreements on double taxation and tax harmonization between the EU member states should actually prevent that. Heck, I even worked in the US once, declared my taxes there and didn't have to do my taxes back home in the EU anymore.
Of course you still need to know your way around local taxes, legal obligations (of the shareholder, of the company) etc. etc. but that's the case anywhere in the world.
> Why are the servers located in Sweden,
We have 762 servers spread across 38 countries. Less than 10% of our servers are located in Sweden [0].
> a country that's known for online surveillance
My cofounder and I started Mullvad as a protest against the growing mass surveillance of Sweden as well as other countries. Our intent was direct political action through entrepreneurship. Incorporating the company in Sweden was the obvious choice, since we are both Swedes. We felt that incorporating elsewhere would have been more risky, complicated, and costly.
Mullvad has excellent lawyers who continuously monitor the legal situation in Sweden. Right now there is no Swedish law that can compel Mullvad to start logging [1]. Should the situation change we have contingency plans.
[0]: https://mullvad.net/en/servers/ [1]: https://mullvad.net/en/help/swedish-legislation/
> Our intent was direct political action through entrepreneurship.
Again, I didn't mean to question your intent – as I said my impression of your company so far has been a very good one! :)
> We have 762 servers spread across 38 countries. Less than 10% of our servers are located in Sweden [0].
My apologies, it's been a few years since I last looked at your server locations (so I didn't remember) and I was probably getting the wrong impression from the fact that your post is only mentioning server locations in Sweden.
> Right now there is no Swedish law that can compel Mullvad to start logging [1].
But at the same time all cross-border traffic in and out of Sweden (so for anyone using Mullvad outside Sweden: virtually all traffic) is being monitored and (probably) logged, isn't it?
OP's been doing a poor job of "following Mullvad for a long time" and apparently has never used Mullvad or visited its website. Heck, it's on their Wikipedia page[1].
If they had, it would be immediately apparent that Mullvad has servers all over the world. They have for many years -- perhaps since it's inception. It takes a bare minimum of effort to learn that.
Sometimes a mediocre question is the landing pad for a great answer. No need to begrudge the question.
That most certainly should be called out -- especially when it was, at the time, the top voted comment.
And I'm not here for upvotes. Who's being combative, again?
What more do you want? Upvotes are not under my control and I also cannot edit my original comment anymore. No reason to get personal.
FWIW, I still think that it's important to raise awareness for the deficiencies of Swedish privacy law (irrespective of Mullvad and where their servers are located). I suspect that at least some of the upvotes were also given because people agreed with that.
I am a regular user of Mullvad and recently wanted to try a different VPN, that only provides Wireguard configs (i.e. no native app). I used the default setup.
For some reason, my internet connection was flaky, and when it disconnected and reconnected, my traffic leaked.
That never happened to me with Mullvad as the app comes with an "Always require VPN" option out of the box and it has always worked reliably.
Plus it gives you a psychological separation between "VPN related activities" and not. Or you just do everything in the VM. Adds a layer security wise as well to protect your physical system.
If you wanted to get really fancy you could have a few different VM's and each one on a different companies VPN
https://github.com/VTimofeenko/wireguard-namespace-service/b...
I'm actually kind of curious about what Wireguard does here. I think Wireguard says it's connected almost immediately even when it isn't, presumably holding traffic back locally while it waits for the connection to be active. I was wondering because I spent some time confused by a non-Mullvad Wireguard connection that wasn't working (turns out the server wasn't available at all) that nonetheless appeared as "connected".
[1] https://blog.haschek.at/2019/build-your-own-datacenter-with-...
For reference, ExpressVPN (which has been audited by PwC) introduced this in 2019 [1].
Unfortunately, ever since ExpressVPN was purchased by Kape Technologies (they also own PIA, Cyberghost, Zenmate all of which do not have reliable histories); Mulvad has been the clear choice for a while now. They're also the backend for Mozilla VPN (mozilla just whitelabels from Mulvad [3])
[1] https://www.expressvpn.com/blog/introducing-trustedserver/
[2] https://www.expressvpn.com/blog/pwc-audits-expressvpn-servers-to-confirm-essential-privacy-protections/
[3] https://news.ycombinator.com/item?id=26646510I live in Kazakhstan and recently our government decided to shut down the Internet. But apparently there were ways to get out: they did not filter two TCP ports. My guess it was some "backdoor" put by employees who had to obey the orders but wanted to provide people some way to get around those blocks. Those ports were used to run VPN software. I used Outline VPN on my VPS and it allowed me and my friends to have a working Internet.
TLDR: allow specifying port and protocol (TCP/UDP) as some kind of advanced option for those users who need it for some reason.
Right now we've got Internet back and it works fine, but who knows when our government will decide to shut it down again.
PS mullvad.net website apparently is blocked in Kazakhstan as well. I know that they block popular VPN provider websites, so that should not come as a surprise, but still. I have no idea whether actual VPN subnets are blocked or not.
When I hear about things like this it makes me glad I have a simple satellite communicator - it will only do short text messages but that is a hell of a lot better than nothing. Of course one could get a full satellite based mobile internet device or phone but the plans on those get quite expensive.
Don't forget to add insta-shutdown when any USB device is connected to the system!
As a result they can absolutely disable USB entirely by never exposing those parts of the device tree to Linux.
https://www.ovpn.com/en/blog/improvement-of-the-physical-sec...
An alternative is you use a RAID array and mount your disks in read-only mode, or use physically read-only disks and when you have to replace a disk, you pre-mirror the replacement disk. In this way the local disks can be replaced as they fail and there's never a point when the server is at risk of not being able to boot.
......or they could boot from CDROM :)
They can still use local disks to provision the OS over a network but boot from local storage, and prevent writing to disks from the booted OS (hell, they can completely remove the disk drivers from the kernel!). It just doesn't make sense to ditch the drives from a reliability standpoint. They're going to have a big outage one day just because they didn't want to deal with drives.
The point of not using local disks is again fairly straightforward, to show that they do not have a stateful storage medium to write logs to. Whether it significantly helps or not is beside the point, they have determined that it helps provide assurance to their customers and additionally showcases a feature for auditors.
Network booting loads the OS into RAM, so even if there was a network outage they'd have to restart the severs to cause a problem. From what I know of most VPN solutions though, again a network outage would only affect the group of servers at that data center which isn't their entire operation.
[1]https://www.perfect-privacy.com/en/features/without-logs [2]https://www.azirevpn.com/docs/environment
There's just no good answer to perfect trust-no-one private internet access.
If you need to hide all of your traffic from other users in your local network, you can accomplish that in a trust-no-one fashion by running your own VPN endpoint on a server you control which provides better privacy guarantees compared to a centralised commercial VPN whose business model will eventually involve selling your data (once user growth stops but shareholders demand continued revenue growth).
But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.
What kind of people?
How do you verify you can trust some company?
You verify trustworthiness by research. Who is involved? Do I trust anyone who trusts them? What are their motivations? What would cause them to take action against me? What causes them to protect my interests? What laws are they subjected to (i.e. who can coerce them)? What do they say for themselves? Where do their words fall on the credibility to BS scale? What is their reputation in the community? What do their competitors/adversaries say? What would cause their behavior to change?
I won't enumerate all my research on Mullvad. I can say Mozilla attaching their brand to Mullvad's services helped me a lot (trust by proxy). I'll also say that some of their product decisions give credibility to their anonymity claims. Lastly, I found someone who shared a competitive analysis across many providers. I found the analysis trustworthy. Mullvad has some weak points, but was still the best provider for my particular use case.
https://mullvad.net/en/blog/2021/9/16/ownership-and-future-m...
A VPN provider can "accidentally" enable logging prior to the sale
What about Tor?
For maximum privacy, Tor should be used with software designed for Tor from the start.
PIA has been promising a fully audited and verifiable infrastructure in the future:
https://www.privateinternetaccess.com/blog/dont-trust-verify...
yes. Which is why I said that this helps to shield your traffic from other people in your current local network (think: coffee-shop) which is one use-case of a VPN.
If you need to protect your traffic from anybody but your peer (another potential use-case of a VPN if this were possible) and you even want to hide the fact that you were talking to that peer, then you're out of luck. Period.
the privacy protection for most people using VPNs is required against their ISP and other actors looking to analyse their traffic, not users on the local network. a commercial VPN will be better for privacy due to the crowding effects, ie. large number of users sharing the same IP and protects against correlation attacks - it's much easier to trace the activities on your own VPN endpoint back to you. of course you need to trust the operators, which is as different question.
Depending on where you are based in the world (see https://www.submarinecablemap.com) realtime throttling of vpn traffic can still identify a user and where they are going in some cases.
You can get a degree of privacy from visiting websites located on servers in big data centres, but nothing a search warrant couldnt find out retrospectively.
Just traceroute your journey inside a vpn to see where abouts you are going when connecting to a webserver anywhere in the world and workout the physical route you are travelling on the cable map.
Obviously the number of languages you speak also restricts where in the world you will be going online to a point and timezones can also make you stand out like a sore thumb if you visit a website when the locals generally arent.
I've identified (US) websites which can workout what DNS server you are using, so in my case, based in the UK if I swap from using a UK ISP dns to using another dns like quad9 in Germany, the (US) websites alter the content you can see, just on that single DNS server change.
There is no privacy!
I think this is a good message. In the same vein, there's no security either. All you can do is make your and your adversaries' life harder, and balance the different tradeoffs.
Dont buy that, care to elaborate?
In the real life, no lock is invulnerable. Most can be picked, frozen, melted, etc and surely have other weaknesses too.
But to achieve their goal, they don't need to perfect. Just reasonably good. And so, I wish for people to be mindful about the nature of these. That they are not perfect, they are not hidder, nor secure. Just, maybe, reasonably so.
In IT, you need both joined up offensive and defensive measures which includes self destruct if secrets need to be kept. That is at best a Check Mate.
Take a VPN, in nearly all instances I have encountered the only traffic is genuine traffic, there is no dummy traffic to muddy the waters from external Deep Packet Inspection.
Likewise routing can be used to isolate, I'll give you a real world example which you might be able to relate to.
You are travelling by car from A to B, and you can take a variety of routes to get there. Most modern cars now have built in sat nav, and all you know is when your target is leaving and they will be using the car manufacturers satnav. So you have a window (at the start of their journey) in which to manipulate the targets satnav by giving it fake traffic data to cause it to take a particular route. Ergo you have been able to isolate your target onto roads they wouldn't normally travel. Now that can be done nationally over the radio station network, or nearby using a transceiver SDR in a chase car.
What makes you think the internet is any difference? Business efficiency like JIT is a weakness as we see with the chip shortages and other problems caused by covid lockdowns. VPN companies are no different, they need to maximise profit so they dont add in fake traffic to hide their customers traffic, and by virtue of being able to choose from multiple VPN providers, users self isolate themselves into yet smaller groups. VPN providers should really organise and share networks to further muddy the waters from external entities.
Well not really. There was a great (german) interview with the perfect privacy founders recently [1]. They seem to be decent guys with close ties to the Chaos Computer Club and I strongly suspect they wouldn't want to work like that.
[1] https://www.youtube.com/watch?v=VMr0gJvI-6I
> But if you need to hide your traffic from anybody but your peer on the internet and you need to hide the fact that you talked to that peer, then, I'm afraid, your out of luck.
Nah, that one is easy just use an anonymous sim card or an open wifi and your good to go.
Honestly these discussions often feel pretty asinine to me. I personally use paid VPNs to pirate to my hearts content, work around my ISPs terrible networking and a little bit of geo-unblocking. Of course you can't use these services to protect yourself from three letter agency type surveillance or equally powerful threat actors but if they are "private" enough to block the music industry and their lawyers from suing you that's a pretty high standard of privacy, certainly more than any ISP alone gives you!
Generally I don't care much about the UI of any of the services offered and being browser based doesn't really make it any better for me. I can do that with spotify and most other services as well. What I would like to have is a simple paid service with high quality flacs that has an open enough API that i can use many of the great open source tools available and download music for offline use on my phone (data caps and all) without jumping through a lot of hoops. It's not music management is an unsolved problem and for local music i have tons of great options on all my devices from TUI applications to applications with great desktop integration to great open source phone apps. With Spotify there are at least some projects that work somewhat but not really well and certainly nothing that i can easily integrate into my desktop or phone without relying on proprietary clients.
But honestly before I go around trying endless services to get a decent experience I'd rather just take the red music tracker test [1] and build a local collection that "just works" and be done with it.
(it was blocked for me because it's a VPN service provider)
Also, isn't there a law that enforces logs to be kept for n years ? How is it compatible with diskless setup ?