We moved our servers to Iceland | Hacker News Reader

We moved our servers to Iceland

blog.simpleanalytics.io

241 pointssoheilpro125 comments

> That’s right, we need to enter the key when the server boots. Wait, but what happens with a power failure? Are all requests with page views to your server failing after a reboot?

> Then we found Dropbear, a very small SSH program, that you can run via the initial ramdisk (initramfs). This means we are able to allow external connections via SSH. We don’t have to fly to Iceland to boot our server, yeah!

Or you can use Mandos to be able to sleep through a reboot: https://www.recompile.se/mandos

Disclosure: I am the co-author of Mandos.

I am not sure that the offset in security is that small. The assumption, that your servers will only be evaluated month later doesnt hold true in most cases. Attacking running machines quickly does happen. If your threat scenarios are little green men, they are unlikely to just shut down your server and haul it to some dark room to interrogate later. They will of course try to get the key from a running system, but thats also something you should most definitely have in your threat assessment.

edit: blue men*. Forgot that green police uniforms are an outdated local thing.

I’ll give examples for the US but you can translate as appropriate.

I’d say there’s no municipal police force in the country that has a sufficiently technically sophisticated regular law enforcement to try to attack a running machine. So if you are suspected of having information relevant to a murder or drug deal—the seize and examine one month later scenario is extremely likely.

A handful of states and maybe one or two municipalities may have a computer crimes division that would be sophisticated enough to know to worry about encrypted disks, but they’d have to hire expensive outside consultants to carry out the actual attack and would do so in exceptional circumstances rather than routinely.

Finally, if your threat model includes the national security apparatus then, yes, this is probably too much of a compromise.

There are regional technical support task forces that work across agencies. Smaller departments call them in for serious work and the larger ones contribute resources. So while you’re right about the agencies’ individual capabilities, you’re wrong that they’re siloed like that.

>I’d say there’s no municipal police force in the country that has a sufficiently technically sophisticated regular law enforcement to try to attack a running machine. So if you are suspected of having information relevant to a murder or drug deal—the seize and examine one month later scenario is extremely likely.

I wouldnt assume that this is still the case, at least in Germany. I think the days of police shutting down the devices they have a search warrant for are pretty much over until they are confident enough that there isnt a disc encryption in place. If there is, they keep the device from going to sleep and call for an expert. If they are actually targeting server infrastructure, there will be people with technical expertise involved.

The US can have some small police forces. Like, a handful of people independently serving a city.

If they were the first to respond to a tech crime, they could probably create a mess. But in theory, they could do their own search warrants, searches, recommending prosecutions, etc.

Even big police forces, e.g. NYPD, have lots of divisions. I don't think there's any standard operating procedure in place that would lead a random detective in a random precinct to call in one of the two or three experts across the entire NYPD on a case he is working that happens to involve seizure order for servers. If someone in the special city-wide child pornography unit was involved in a server seizure, that's when the computer expert might be involved (because he or she would be attached to the computer crimes division the child porn unit is a part of).

> They will of course try to get the key from a running system, but thats also something you should most definitely have in your threat assessment.

1. I have yet to see any indication that this is the case.

2. If you assume such technical skill in your adversaries, you must figure that they could just as well open your servers and read the file system keys right off the memory by running wires to the memory bus. In which case Mandos is no worse than no Mandos.

Putting wires on the memory bus is not really needed.

>1. I have yet to see any indication that this is the case.

Taking memorydumps of running systems have been state of the art for about 10 years. There is a heap of commercial forensic software which are used for extracting everything from bitlocker to back then truecrypt. In 2014 it was published that the German police used ElcomSoft as well as Passware, both capable of extracting keys from running machines. I doubt you even need experts for that anymore. Police officers were already on alert to prevent you from shutting down your devices years ago in cases where they assume that there might be evidence on them. I mean we are living in a world where the police is cloning smartphones in routine controls. And on the more sophisticated site, where the police is allowed to infect suspects of even low level drug dealing with trojans. The police isnt stuck in the 90s and they are aided by a rather large sector producing forensic software as well as spyware.

I understand that your product offers great convenience, however its only reasonable to point out, that the need to put in your password after the device was shut off is not a problem but working as intended. Encryption will only protect you if the device is turned off. Circumventing this is not necessarily a good idea.

edit: You also dont have to assume law enforcement. The tools are also affordable for private entities and marketed towards for example towards private detectives.

Even if your adversary have access to the latest and most sophisticated state-of-the-art tools, and have the budget for having technicians to use them, Mandos still does no harm; i.e. does not reduce security. And Mandos is not meant to defend against this; Mandos is instead meant to get people who are not using encrypted disks to start using encrypted disks, since Mandos fixes the (in my opinion) most annoying problem with using encrypted disks.

You are, in effect, arguing about whether encrypted disks are useful at all, which is a debate I’m really not getting into at this time, since it has nothing to do with Mandos.

>You are, in effect, arguing about whether encrypted disks are useful at all

No at all, encrypted disks are very useful, you just cant show anyone the content.

The sole purpose of encrypted disks is to prevent somebody else from reading them once they are turned off. Your product turns them back on and makes it readable again without your interaction. Turning off those devices is however the only protection you had. So now you also need to turn off a second server.

And once someone has access to your running machine he can read it. Be it with simply copying the files or getting the key via some software with a big "HACK" button. https://www.elcomsoft.com/efdd.html

> Turning off those devices is however the only protection you had. So now you also need to turn off a second server.

I’m not sure I follow. If I understand you correctly, you want to have a panic button to press in case of physical intruders, and without Mandos, the power button of a server served this function, and your’re worried that Mandos makes you lose this button? Not to worry. The simplest solution is to have two local servers be the Mandos server for each other, each enabling the other to reboot unattended, and the panic button would be the power button on the power strip powering both servers. Once both servers are off, the system is effectively locked.

If you insist on having a remote Mandos server (which is not really the use case it was made for, but it is supported), you could always automate some button to signal the remote server to disable (or outright remove) the client, thereby denying all access to the secret password. The Mandos server process is controllable via D-Bus, so any program can be made to signal the Mandos server in this way.

With a programmable power strip you could even have a single button doing both things. So there’s your panic button back.

> And once someone has access to your running machine he can read it.

That has nothing to do with Mandos. Anyone with physical access to a running machine can already, in theory, access the memory and get the encryption key. Mandos introduces no additional threat from a theoretical sophisticated attacker.

There is several parts to this, depending on the threat model.

One is the scenarios where both machines are turned off during theft or fishing expeditions from legal enforcement. In this case running encryption, any encryption, is better than no encryption, and Mandos allow the administrator to install it once and forget.

I take it that your concern is that someone has turned off a machine and forgot the Mandos server, and then a time later a attacker has capabilities to break into the encrypted running machine through unrelated vectors. Mandos do not protect against attackers that has capability for breaking full disk encryption on running machines.

The threat model where we assume the attacker can break into an running machine that is encrypted looks very different. Devices that has mitigation against this tend to have in my experience things like movement sensors, light sensors and GPS, with plenty of kill switches that fry the internals if anything look at it funny. Those kind of threat models are fun, and I enjoy talking to those kind of engineers that work on such servers, and to them I don't recommend Mandos.

Where I do recommend Mandos is to the administrator that for reasons are not physical near the server that they maintain. Maybe they work at multiple buildings or travel. They have servers with sensitive services which won't be turned off for long periods but still need reboots once in a while, and yet goes unprotected because there is no one available to write in a password at the physical location.

A other scenario is when people want to treat a location as safe and want protection in transit. In that setup the client can be treated as any unencrypted device for all purposes except when taken outside the local network, at which point it behave like an encrypted device where the person carrying might not even have access to the password. In theory this makes for a great method to transport a work laptop between offices in different countries, booted into an throwaway operative system.

As with any security, thinking hard about the threat model is essential. What are you protecting, who is the attacker, and what are the risks.

Disclosure: I am also a co-author of Mandos.

Re your edit, I always thought lgm was a reference to military garb, not police. Also, many police wear black now.

Makes sense, things got really butchered in the translation. Our police used to be green until, what feels like really recently. So there are still a lot of sayings around involving "the people in green". Now they are apparently blue across the EU.

To me, "little green men" meant aliens.

“Little green men” to me is a reference to the Russian army units that invaded Ukraine without insignia for deniability.

Thats a rather recent one however. The saying is rather old.

Usually, but in this context it's clear that it isn't.

Wow, that looks really cool. However, I use CentOS 7 and that documentation appears to indicate it's not an officially supported distro. Are you aware of alternatives to Mandos (even if they aren't as good) which have official support of RHEL/CentOS?

Sorry, no idea; haven’t heard of any. Mandos is made for Debian for the simple reason that it’s what we use ourselves, and at the time Mandos was created, only Debian had something even approaching the flexibility of initramfs-tools. Nowadays with dracut, it should be possible to port it, but we haven’t looked at it.

On first glance this looks great and solves a real problem. Not sure how I’ve missed this.

Is there much of a plug-in ecosystem for this to support various auth backends?

*edit grammar

I’m not sure what you mean by “auth backend”. By default, the client merely has to prove that it possesses the correct key to be given the secret encrypted password from the Mandos server. This can be changed by configuring the server not to automatically “approve” clients, which makes the whole procedure controllable over D-Bus, opening the way for any process you like to do whatever it needs to do in order to authenticate a client before the secret encrypted password is sent.

Why does the root need to be encrypted? What's wrong with an unencrypted root and /var and /home and swap and /mnt/storage being on encrypted partitions/zpools?

I boot my machines and then ssh into them with a script that pipes a local gpg -d to them to unlock their disks, mount appropriate filesystems, and start necessary daemons. Unless the daemons are poorly-behaved, no data that is specific to me aside from the host's SSH key is ever written to the root - only the OS/distro files.

I thought this is what everyone does.

The root file system still, at the very least, records access times of all files, so an analysis of the the root file system releals, for all programs, when you last ran them, and for all libraries, when they were most recently loaded, etc. With Linux defaults, IIRC this timestamp has a limitied granularity of a day, but this can be telling enough.

I thought most people were mounting things noatime for performance these days.

The noatime mount option is not the default, so no.

That will work if you are sure that all the sensitive information you want to protect goes exclusively to the encrypted file system. Which is a tall order and security headache by itself.

Was fairly leery of how useful I thought this would be, but you reference the FAQ[1] early in what I assume is an attempt to head this off, and it delivers adequately. Despite being predisposed going in thinking it was probably not worth the janky stuff you would have to do to get it working, the level of detail and thorough covering of some fairly complex attack vectors leaves me thinking that while not as secure as no automatic boot method for encrypted disks (obviously), it would definitely be a step up over not encrypting and something I would recommend without worrying that I was increasing the burden of managing the systems to a level an organization would be unprepared to deal with.

Not that I'm some professional security researcher or anything, but congratulations, color me impressed.

1: https://www.recompile.se/mandos/man/intro.8mandos

It's a little sad to watch someone jump through hoops to try to please the people behind EasyList, when I sure don't see anything about country-of-hosting in EasyList's policies. Especially when almost nobody actually cares about the other members of the Five Eyes or the Enemies of the Internet in these discussions, but just knee-jerks about US hosting.

Yes. Easylist/EasyPrivacy here is intended to block ANY third party tracker. This is a tracker. I use this adblock list and I don't care what country you're hosted in, simpleanalytics is a third party tracker and should be blocked.

> " EasyPrivacy [...] including web bugs, tracking scripts and information collectors"

Simpleanalytics is all of the above. There is no arguing this.

I know very little about SimpleanAlytics. However, they do claim to not be a tracker: https://simpleanalytics.io/no-tracking. So, they at the least, would argue one of your claims - do you have evidence that what they are saying incorrect?

By definition, loading it will send your (regardless of whether or not they save this or associate it permanently on their end) coarse location via IP, OS, CPU architecture, browser and via user agent, referrer, execute code on your machine, pages you visit.

In their page they say (w/r/t Google Analytics) "you maybe use the anonymization feature. This is a step in the right direction, but the sad reality is Google can still collect all the information" -- you would have to trust them too [Simple Analytics] to not collect, save, or delete-accordingly all the information sent to them.

It is a third party information collector, period. It tracks visits, it tracks pageviews, at minimum. I am not against them as a company or product, but I believe that easyprivacy is 100% correct in blocking their domain.

I think its a big mistake to lump this type of "tracking" in with more involved "tracking" where particular users are followed around from site to site to serve up ads based on what the user has looked at recently. These are massively different things. Maybe you are opposed to both types of tracking - I dunno. Personally, I have no issue with a website keeping track of the things that they claim to keep track of (https://simpleanalytics.io/what-we-collect). However, when tracking means that a user does a search for something like "maternity" on one site and then starts getting ads for baby diapers on another - that feels like something that I understand why someone would want to avoid.

Even if I kind of agree in principle, what you are asking me is to trust what the provider claims it will do with this data. It’s an industry that I just do not trust, even if I had the appetite to do the due diligence which I don’t. Even if I knew the guy and was convinced he had good intentions, he might still get hacked or acquired and the data used some other way in the future.

Therefore I expect my blocker to block a 3rd party tracker which by definition has the capability to track me across websites. It makes that kind of tracking technically infeasible which is the level of protection I want.

We can bikeshed about the peculiarities of tracking all day long, but I don’t want any kind of tracking. Period.

I already provide all the necessary data in my request to the first party server. It’s all in the server logs.

I agree, there's tracking and there's counting. SimpleAnalytics appears to just count hits, but doesn't track users - no cookies set, no browser fingerprinting. Sure, we have to trust in them really not storing IPs etc, but it does look good. Not tracking users, only global site usage.

Sure - I see nothing wrong with separating out the categories. There are different types and different levels.

End of the day, at least in my opinion, it is acceptable to be added to blocklists, and it is acceptable to not use those blocklists and hit the beacon.

By that logic any external resource request including images and JavaScript must be blocked.

Yes. A website can host everything it serves. Including 3rd party stuff on your page is sharing data about my visit with 3rd party. So I'll block all 3rd party stuff by default, with manual exceptions

> There is no arguing this.

Yes there is. SimpleAnalytics claim that they "don’t track visitors of our customers’ websites".

Are they lying? If not, how is it a tracking script?

I guess they mean that they aggregate data? rather than storing it for individuals users?

Otherwise, why do they need your browser to make a request in the first place?

FWIW I agree with dylz. It is crazy to suggest that any company doing analytics has any sort of right to make my software connect to its servers.

If Easylist did not block all unnecessary requests, I would find a list that did.

> It is crazy to suggest that any company doing analytics has any sort of right to make my software connect to its servers.

No analytics company has a right to make you connect to its servers. But, that isn't the case at all: You visit some website. The operator of that website has contracted with the analytics company. You ask the website for some information and it replies with that information as well as a request to ping the analytics company that the website contracted with. It's apparently "crazy" to comply with that request, all while still consuming the information that you asked for and received.

Ad-blockers wouldn't be controversial if they worked by navigating away from any site that displayed ads or included a tracker. Of course, that would be inconvenient for the user, so, they don't do that. The whole point of an ad-blocker is to allow a user to consume a service in a manner that the entity running and paying for the service didn't intend. We could at least acknowledge that. Your position, however, seems to be that doing what the service you are using is asking you to do in exchange for its information is "crazy" - and that is straight up ridiculous.

I understand your viewpoint, but I consider all script tags that a server provides to be merely suggestions.

A browser is, after all, a user-agent. The web was designed this way for a reason.

Would I be breaking some sort of inferred agreement if I browsed with no Javascript enabled at all?

In any case, a site server logs should provide ample information for analytics, unless they are measuring mouse movements or scrolling etc.

Just wait until that official DRM crap is used for adtech and preventing you from modifying the HTML the server sends.

This is a war; a never-ending war of users VS adtech. I realize how much defense I have to run to protect mtself even moderately, and I'm still losing.

> Ad-blockers wouldn't be controversial

Adblockers are not controversial. Stop pushing adtech narrative and passing it off as an established fact. HN is the last place on earth where people would fall for this kind of crap.

The internet was built to decentralise and make information globally available. It wasn't built for some morally bankrupt interest groups to turn a profit. The http protocol is explicitly in favour of user control over content.

I get to decide what runs on my hardware. There is nothing ridiculous about that.

Ad blockers are not controversial, even if the ad industry would like them to be.

> Ad-blockers wouldn't be controversial if they worked by navigating away from any site that displayed ads or included a tracker. Of course, that would be inconvenient for the user, so, they don't do that. The whole point of an ad-blocker is to allow a user to consume a service in a manner that the entity running and paying for the service didn't intend. We could at least acknowledge that.

It's the other way around. The HTTP protocol and supporting web standards were designed from grounds-up to allow and encourage the kind of things an ad-blocker does. The browser acts on behalf of the user (hence the term "user agent"). Links in a HTML response are information, "there is something related over there", not a command "you must go there"[0].

The right way to solve ad-blockers is for sites to comply with the protocol they're operating under - to refuse delivering a resource, with 402 or 403 code, until payment is provided, or an ad is displayed. AKA "the paywall". The controversy only exists because many website operators prefer to be dishonest and manipulative - they post content that they mark as free giveaway, but simultaneously demand compensation. They stir up drama of how ad-blockers are immoral, whereas in reality, blocking ads is "playing by the rules" and it's them who are in violation of human decency.

> Your position, however, seems to be that doing what the service you are using is asking you to do in exchange for its information is "crazy" - and that is straight up ridiculous.

It's not crazy. But it's also not required by any technology, law or custom. Thing is, the service is asking the wrong way. HTTP protocol was created with means for asking to do something. Like, by responding with 402 Payment Required or 403 Forbidden and some instructions on what you want the user to do, instead of responding with 200 OK + content + guilt-tripping popups and pretending to be victim in news articles.

[0] - Want the "command mode" web? Invent your own, DRMed one. Because otherwise what you're doing is, again, trying to have your cake and eat it too - putting your content on the public web to gain free audience, and then refusing to play by public web's rules.

How do you feel about older screen readers that don't handle JavaScript well and probably won't load the ads (which probably aren't accessible anyway) to read them to the user? Has the user "stolen" or simply used their _user-agent_ to do what is best and most accessible for them?

> The whole point of an ad-blocker is to allow a user to consume a service in a manner that the entity running and paying for the service didn't intend. We could at least acknowledge that.

It's kind of funny that that was, originally, the very definition of "hacking", and we are in a site called "hacker news". But moving on...

The adtech industry got in its head that it can just arrive in a place that already existed (the Internet) and start inventing implicit contracts. It reminds me of the "nice guy" that does some nice gesture for a woman he likes and then gets outraged when the "implicit contract" for sex is not honored.

Hosting is not even that expensive. You know what's expensive? "SEO", buying traffic, in short polluting the very environment where you exist. That is the sort of thing many companies will do with their ad money. They do what they can to place themselves between me and the stuff I want to talk/read about, and then act all outraged when I take counter-measures against their spying and manipulation attempts. That's rich.

I hope EasyList continues to do its job of acting on behalf of the users and does not get swayed by bullshit.

> I hope EasyList continues to do its job of acting on behalf of the users and does not get swayed by bullshit

They often bend over though. I don't remember specific one, maybe amp-analytics.google.com. Also removed some anti-adblock thing after DCMA complaint.

You have to use several lists, like uBO does.

They can both be right.

"Tracking" has never been very well defined. It just means something like, "collecting data we don't think you should collect."

If you want something more specific, lots of conflicting definitions do exist, but nothing that people can agree on.

It is a 3rd party collecting data about my browsing history. Give me one reason why I should not block it, please.

They don’t track customers, and they respect ‘do not track’. One of these must be wrong.

I think it would be useful to differentiate between “user trackers” and something like a “site tracker”.

A “user tracker” sets cookies and follows a user around, knowing what the user have or have not done before.

A “site tracker” (bad wording I know) follows what happens to a site, what device types accesses it etc.

Sure, making direct requests to the analytics servers exposes IP and possibility to somewhat track the user. Tracking based on IP is pretty inefficient though, due to shared IP-addresses, IP-address changes when jumping between networks, VPN:s.

Ideally SimpleAnalytics would provide a tiny proxy/washing script that one can host on ones own server and which one includes on ones site rather than including it directly from SimpleAnalytics servers. That way one can guarantee ones users that no IP-address is leaked to a third party (which would also make a Data Processing Agreement completely unnecessary from a GDPR perspective)

I think it's unfair to assume he's jumping through hoops to try to please EasyList when you could just as well assume that someone from EasyList highlighted an issue worth dealing with, even if he knew that there was 0% chance SimpleAnalytics would ever be whitelisted.

> If you draw a straight line from San Francisco to Amsterdam you will cross Iceland. Simple Analytics has most customers from the US and Europe, so it makes sense to pick this geographical location.

1. I would be interested to find out if being geographically between San Francisco and Amsterdam is actually good for latency.

2. I think the usual solution to having customers in the US and Europe and wanting to keep latency down, is to setup servers in the US and Europe. So, this strikes me as an odd justification of the decision.

Pings to 1984.is:

From Palo Alto : round-trip min/avg/max/stddev = 183.514/184.035/184.898/0.466 ms

From Amsterdam : round-trip min/avg/max = 36/37/39 ms

From London : round-trip min/avg/max = 48/48/50 ms

Obviously this can vary quite a bit depending on what peering you have at your disposal.

For reference, from Palo Alto to London: round-trip min/avg/max/stddev = 156.236/156.921/158.271/0.834 ms

For latency, I would choose NY/NJ or similar if you really want one location to serve both EU and US; Though this isn't ideal, it does lower the latency to West Coast US quite a bit.

From San Jose to Newark : min/avg/max/stddev = 73.462/73.547/73.660/0.078 ms

From London to Newark : min/avg/max/stddev = 72.836/74.323/75.600/1.200 ms

From your pings, it seems pretty likely that traffic from your Palo Alto location is going through europe to get to iceland, most likely through London.

The submarine cable map shows one cable going west to greenland and then Canada, one that goes to northern England, and two to Denmark. The Canadian landing isn't very close to any other transatlantic cables, so it may not be very well connected (land connectivity is much harder to map though).

Anyway, making location decisions based on assuming internet distance is similar to physical distance is kind of silly. There are many physically adjacent countries that don't have direct interconnects, it's pretty common for traffic to exchange through somewhere much farther away than the ultimate destination: many south american countries exchange traffic in Miami.

You’re right that it’s silly - from an IP topology layer Iceland might as well be mars. There was a bunch of news several years ago that Iceland was awesome because geothermal but a) there are other grids that are just as good if not better (Norway for ex) and b) doesn’t erase the otherwise terrible trade offs in network and dc options that are available.

Are these all your employer's servers? Or is there a service you utilized?

Looking glasses from Leaseweb and Softlayer, which are very well connected to private and public peerings.

Sadly, packets do not follow great circle paths to their destination, but have to travel inside of existing extremely expensive and extremely long fiber optic cables, laid down by extremely expensive ships custom-built for the purpose.

On which map projection?

Probably great-circle distance.

Matomo is analytics for folks who really care about privacy (or folks who want to simply have more control over their users analytics data). Install https://matomo.org/ and run it in the same location as the project you're already running. Your own instance of Matomo is not going to be on EasyList any time soon.

Matomo is blocked by many ad blockers by default, not based on the domain name or URL but based on the name of the js files downloaded by the browser.

...and those could be easily avoided by referencing "js/" instead of "piwik.js"[0], or by running Matomo's log analytics tool[1].

Which leads back to the following quote from the author:

> But what happens if we block alternatives even if those alternatives are taking the privacy of the user very serious. I care about the privacy of the individual. I don't collect any personal information (I don't even store IP's). Even if you have your Do Not Track-setting turned on in the browser I do not collect any information (see our script).

[0] https://ericmathison.com/blog/bypass-ad-blockers-and-track-y...

[1] https://matomo.org/docs/log-analytics-tool-how-to/

I appreciate the intent, but I think this is folly. The FBI has imaged Icelandic servers, like Silk Road's, with the cooperation of Iceland's law enforcement. Iceland's ISPs are also not neutral and have been pressured by other countries to block access to websites like Pirate Bay, which would be beyond the pale for US ISPs.

There is no country on Earth that will meet your requirements for hosting. For example, if you host in Russia, you probably can evade the US government's prying eyes, but then you have to deal with the Russian government's prying eyes.

I strongly doubt Digital Ocean would give customer data to anyone without a warrant from a judge. And there could be some scenarios I'm not thinking about, but I also doubt a judge would ever grant a warrant to collect bulk analytics data. And I think it would be unlikely that law enforcement would want to request a warrant just for some narrow analytics collected on a few specific individuals.

Also, as others have pointed out, you actually make yourself totally fair game for US intelligence agencies by being in a non-FVEY country, and even more so because Iceland's ISPs peer with ones in FVEY countries.

But way more importantly than all of that, the threat model is wrong. You're likely at far greater risk from cybercriminals and regular blackhats than you are from any government. Digital Ocean (very much unlike Linode and some other big providers) has never had a (known) breach, and probably invests way more into security than the Icelandic provider you switched to does. DO likely has many world class security engineers employed; maybe your Icelandic provider does, too, but it's less likely.

And this isn't even going into the added management and latency issues.

I feel like you're kind of handicapping yourself without any significant privacy gain or increase in customer acquisition. You're getting feedback from very suspicious people who want to block all use of your, and others', services. You should take feedback from such a group with a heavy grain of salt.

This also does nothing to actually get the domain removed from the block list - there is probably nothing you can do there, other than gray hat stuff like constantly rotating domains and IPs, or pivoting and changing your entire business model and company.

> I strongly doubt Digital Ocean would give customer data to anyone without a warrant from a judge

A subpeona from a grand jury or an NSL would be more than sufficient.

> We also delete the credit card and email from Stripe (our payment provider).

The email isn't actually deleted - Stripe's logs will permanently hold that data. If you don't want to retain the email address, you'll need to send receipts etc yourself.

Ah, good point. We already do so it's just a matter of not sending the email address to Stripe. Thank you for the suggestion. Added this to our roadmap: https://simpleanalytics.io/roadmap#109207

I applaud the intention but I doubt this is going to make a difference when it comes to signing or keeping users.

If the service keeps growing they will quickly realize that latency and availability matters. Uptime would be now a major concern. If something blows up in whatever facility it's used for these servers they will be fucked. No way to shift traffic to another server.

And let's not even talk about computing needs. At scale, that matters. Having managed services also matters and helps reduce operational cost (a lot). Having a bare server in someone's garage in Reykjavik is the opposite of scaling. It's literally a recipe to deprive yourself from the technologies that can make you successful and definitely a way to slow down your progress.

If SimpleAnalytics' goal is to stay small, then maybe this could make sense, but that for me makes this business pointless and it doesn't sound like the vision the founder has either.

Or maybe this is just an MVP until they can have their own on-prem infrastructure with fully encrypted hardware. Idk, maybe the founder's vision go as far as that, and I can't see beyond the downsides of a such a random move.

No you’re right - very little if anything was accomplished here. The other side of the cables connecting Iceland land in five eyes territory and cost like n times more for capacity (and are old etc).

This reasoning is backwards. Host a service in the US and it is at least protected from arbitrary monitoring by the NSA by US law. Host it abroad and you have no such protection.

I think protection by the NSA is a though sell nowadays.

I don't think that's really the sell. The sell is that hosting within the US is the only protection you can have from NSA, since they have a total, unencumbered free hand when it comes to services hosted outside the US, without even a legal formalism to stop them. Like all national SIGINT agencies, breaking into everything outside their own borders is literally their chartered job.

This is protection against the NSA. My point is that within the United States, there are at least some legal and procedural protections (however flimsy you think they are) which do not apply to a service hosted abroad.

Isn’t the key on the VPS’s memory? What’s the use of this full-disk encryption?

In practice, it is harder to dump the memory of a bare metal server than it is to simply yank out a hard drive. (It is, of course, possible - the technique is beyond many hosting companies and local police, and the effectiveness depends on the temperature.)

In theory, the hosting provider, having physical access, has both the key (in RAM) and the cyphertext on disk, so logically there is little point.

It’s worth noting that my hosted bare metal boxes have encrypted data partitions with keys I provide over ssh each boot.

Of course, if it is a VM, anyone with root on the hypervisor (i.e. the hosting company) can trivially dump the memory and encryption keys.

Encryption at rest. So, if a hard disk (including a turned-off laptop) is stolen, mishandled, or not destroyed properly.

Someone physically attacking a running computer is a different, and much harder, threat model.

This is why companies employing fde will also require laptops to be shutdown before being moved.

How did you choose between Iceland and Estonia, what were the pros and cons?

Iceland has 100% green energy and their location is slightly better. Maybe the biggest plus is their laws.

And also it's a very cool place to visit :)

> We are thinking of setting up a very simple CDN with encrypted servers, which only serve our JavaScript and store the page views temporarily before sending it to our main server in Iceland.

For the JavaScript, just encourage your clients to copy the script and serve it from their own server with the rest of their files.

I really liked this :)

Thank you for existing.

and all of this had zero effect on revenue and customer retention, congratulations you played yourself

also the FBI has merely made suggestions to Reykjavik Police to infiltrate servers and they did

> After the initial revelation of the server's location in a data center in Reykjavik, Iceland, the filing explains that Reykjavik police accessed and secretly copied the server's data. As agents of a foreign government, the prosecution argues, they weren't required to seek a warrant from any US authority.

The owner of those servers is in US prison right now serving a double life sentence

Iceland's domestic laws didn't help here, and the lack of a formal arrangement with cross-sharing intelligence communities didn't help here

So you are either implementing a zero-knowledge service to begin with, or wasting your time

Where are you getting that quote from? I don't see it in TFA, and you don't provide any reference.

If the server uses full-disk encryption, and if it's well locked down, it would be nontrivial to secretly access and copy the server's data.

I mean, adversaries could attach a keyboard and monitor, but they couldn't log in. And you can even delete the root password, and allow only key-based login via SSH.

source of quote: https://www.wired.com/2014/09/the-fbi-finally-says-how-it-le...

additional: https://nakedsecurity.sophos.com/2014/10/10/fbis-warrantless...

> If the server uses full-disk encryption, and if it's well locked down, it would be nontrivial to secretly access and copy the server's data.

OP's article mentions this, part of the reason they move out of the US is because RAM can be trivially read even if full disk encryption is used. Reading RAM still works in Iceland.

Thanks.

> RAM can be trivially read even if full disk encryption is used

I wouldn't say "trivially", but yes, it can be. But if you're that paranoid, you can embed key parts of the motherboard in alumina-filled epoxy. Its thermal conductivity is good, and you can add fins and fans as needed. You can even embed trip wires in the epoxy, to trigger system shutdown if tampered with.

They're running on VMs, not bare metal, presumably, because their (new) server's reverse DNS is vps-*, and previously on Digitalocean. You can just dump the VM's memory space while unlocked, can't you?

A lot of this seems like security theater, especially while still hosted behind Cloudflare.

Oops.

Yeah, here it does seem security theater.

But still, it was a good writeup. I mean, dropbear and all.

I have no clue why they're using VPS, after all that. I mean, if they're a real business, they ought to just setup a server, and ship it to Iceland. If the want the ease of VPS, it's easy to do secure KVM in a FDE server. Even with Docker containers within KVM, if you like.

“If they’re a real business” - this is the sort of dribble HN is reduced to? A real business can’t run on a VPS?

It's not that a "real business" that talks about FDE, and has moved to Iceland for better security, can't run on hosted VPS. But they're being disingenuous if they do so.

But what I mainly meant is that a "real business" can afford to build secure servers, ship them to Iceland, and send trusted staff to set up and configure them.

Ok cool. Now I understand very clearly. Thank you.

I believe the point here is that they claim that they care about security, while their Icelandic VPS hosting provider can just dump the host server memory, which would include the encryption keys.

Then can’t we say that? “If they truly cared about security they wouldn’t use a VPS”. It just rubs me the wrong way the way it’s worded.

Yes, I should have been clearer. Sorry.

This is a good point. We are moving to a dedicated server to resolve this issue.

Nice :)

You probably know this, but anyway. If you're setting up FDE with dropbear on a remote server, it's best to build the installer on the machine.

Evil mastermind, supervillains level :)

I dig around. So to remotely unlock LUKS, on bare-metal in datacenter, one can use custom initramfs, like https://github.com/mtth-bfft/dracut-dropbear-unlock

But still "boot time SSH server's key is stored unencrypted, man-in-the-middle attacks can also be carried out to recover the encryption key at boot time". Ideas?

The full quote:

> Full-disk encryption doesn't protect you against someone with physical access to the machine: the encryption key can be recovered from RAM at runtime (e.g. cold boot attacks), and since the boot time SSH server's key is stored unencrypted, man-in-the-middle attacks can also be carried out to recover the encryption key at boot time. Consider this in your threat model.

So yes, for "Evil mastermind, supervillains level" you gotta embed the RAM, and everything that would give adversaries physical access to the RAM, in alumina-filled epoxy. Or better, alumina-and-fiberglass-filled epoxy. The alumina gives you heat condictivity, and the fiberglass strength. Plus embedded trip wires to nuke the board if they're cut.

It's also prudent to disconnect all ports from the board, except for the NIC, and embed everything that could be used to reconnect them.

With that, it'd be very difficult to get anything from RAM. Attempts at physical access would destroy the system.

But is the storage hdd/ssd epoxied too? The ssh keys to do MITM are in cleartext on boot partition... FBI can put their box on your server's IP, and you'll happily connect, same fingerprint, and type unlock and passphrase.

Sorry. Wasn't thinking clearly. I guess that you'd need to embed the SSD(s) too.

But I wonder if there's a way to reproducibly generate SSH key pairs from hardware IDs. That way, there'd be no keys stored in /boot, and they'd be generated at each boot. So only your hardware would have the right keys.

This TPM guide describes TPM for LUKS. "PCR 0-7 Measured by BIOS" seems like HW fingerprint. https://github.com/morbitzer/linux-luks-tpm-boot

If this thing works, there is no need for remote ssh FDE unlocking.

Thanks. If I understand that correctly, the LUKS key is stored in the TPM. Which is protected by a password. So you boot into initramfs, and provide the password.

And this could also work via dropbear. There'd still be an exposed dropbear SSH key in /boot. But it'd be the TPM that you're unlocking. And I assume that there are ways to verify the authenticity of the TPM via initramfs, before providing a password.

Am I understanding that right? It does seem more complicated. Especially given that you don't have physical access to the server.

> .. the LUKS key is stored in the TPM. Which is protected by a password.

Yup. Nope. The key is protected by TPM magic :). TPM measures stuff, then we put the key in TPM and seal it to PCRs 0-13 measurements (whole boot environment). At boot, TPM will allow tcsd daemon (part of initrd) to read LUKS key once, and only if all measurements match, this is as far as I got.

End result is unlocked volume, without any passphrase prompts, neither on console nor via ssh.

Anyone doing this commercially?

I have no clue. Stuff like that has been standard in the banking industry for decades. I've done it in my workshop.

is anybody doing a solution for this?

like some software that floods all the RAM and memory caches with erroneous data, and then the hardware solution of the epoxy trip wires.

Whenever LUKS is decrypted, the key must remain available. Otherwise, the system can't access anything. If you search about recovering stuff from RAM, you'll find some approaches for obfuscating LUKS keys. But ultimately, they (or the way to access them) must be in RAM somewhere. I think, anyway.

For the trip wires thing, you can just have them apply 5VDC to the wrong RAM pins. And then, if you wanted to recover the motherboard, you'd need to chip out stuff, and replace it. But it's probably only the CPU that'd be worth recovering. And maybe you'd want to embed that too, just in case.

> ..embed scripts .. hosted via the CDN of CloudFlare

I'm browsing to med.com/somedisease and now CF links my browser to the URL from Referer header.

This can be mitigated by https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Re...

This seems to work for links, but does it work for when you embed a script?

> [...] by using the referrerpolicy attribute on <a>, <area>, <img>, <iframe>, or <link> elements

Hm, seems not to work for scripts.

Why not have them on a boat, so you can ship them wherever its cheapest to run them or they are needed? Also ocena provides free cooling water.

Just for the notice: you keep using "we" through the article, yet your home page tells that you run it solo.

The royal we.

Also makes it easier to hire people if you don't have to update any texts ;)

Yep. But it's important to be consistent even for a royal person.

I appreciate what simpleanalytics.io and 1984 is trying to do but I really think they risk going on a hyperbole and thus lose the broad appeal of their cause.

The important thing here is rightful skepticism of companies which gives away valuable complex software (a web analytics tool) for free in return for access to your customers behavioural data which then is sold or used in other parts of their business (ie. Google Analytics/Google Adwords).

Making this a thing about the mass surveillance of the five eyes and thus having to avoid hosting companies such as Digital Ocean and AWS just looses a lot of us and frankly is a bit too paranoid/silly.

A little note on hosting companies. In Europe at least there are corporations that only host data in Europe and don't use any hosting companies from the US. This is a question which we repeatedly get at Simple Analytics. So avoiding those companies has advantages business wise.

What exactly is the problem in using digital ocean and select a server placed in Europe?

The concern is that the US government will ask/force US companies to hand over data or give access even it's stored outside the US. E.g. see the "CLOUD act".

US law enforcement agencies have the right to come to your office, demand and collect all your data and force you to keep silent about it.

You guys in USA might not hear a lot about it but this is a business concern in a no small amount of countries around the world. Especially Europe (or I might think so because I'm there; selection and confirmation bias are always a possibility).

Reply on news.ycombinator.com