Hacker News Reader

If you are proxying the content with a VPN with a static IP on an EC2 instance you cannot police people sending out spam as you cannot see and meter the SMTP traffic. So does everyone get a dedicated instance and IP? If that is the case people are going to have issues getting their email accepted by almost all providers due to the IP being new and on a cloud provider block. If it is a subset of IPs that you create a reputation for and comply with DKIM, SPF etc how are you going to keep it from getting ruined by bad actors?

I am glad you are doing this because running a mail server is non trivial. I have done it for a long time now and love the fact that I own my email identity. It is one of the few things left you can own online.

I think this is the biggest problem. If there are a lot of bad actors, you drag down the whole system, but if you try to prevent bad actors you run into a lot of issues.

A hard problem, and I don't envy anyone trying to solve it.

No offense, but given that you are making a product that has a much higher potential to enable bad actors than usual, isn't it kind of you/your company's job to try to solve it?

EDIT: Just realized I responded to the wrong person :(

No, it's not. In fact it's your job to try and police yourself. But you probably on someone else to control you don't you?

I mean, my thought is that if a high volume of spam causes other email servers to block Helm, then it makes it unusable to the good actors in the system. I didn't see the CEO address this.

Or, maybe I just misunderstand smtp idk

Yeah, that's how I understand the issue too. I don't think there's an easy way to do things, either you end up blocking some people with a legitimate use case or you end up becoming a spam farm.

The person you are replying too isn't the CEO. Just another new user (that's why the username is in green).

You learn something new every day, I always assumed the green meant OP, like how reddit uses blue. I didn't even notice the change in username Thanks!

Thanks for teaching!

Spammers can already get mail servers without a problem. This doesn’t make that easier, because it’s already as easy as it can be.

Hi graybolt, I'm Dirk, co-founder and CTO at Helm. Each Helm Personal Server is assigned their own IP address. We make sure to only use IP addresses that haven't been put on blacklists. If people abuse the service by sending spam it will only affect the reputation of their assigned IP and won't cause harm to the reputation of other Helms.

How do you plan to protect users from themselves? I.e. 2 charater passwords.

If you're spending $500 on something to help keep your stuff private and secure, you're probably not a 2 character password kind of guy.

Oh boy, I know people with a fiber drop, 25tb raids with baby server farms and single character password. I think you highly underestimate the lazyness and/or stupidity of people. That doesn't even cover fishing.

Secondly, I think you underestimate the time intensive work that goes into clearing up an IP. I've run mail servers with users in the thousands. It's basically a full time job to keep a single ip clean. And that's with a half or less percentage of clueless users. I'm unsure how this will scale to hundreds of IPs let alone the thousands(x100) that would probably be needed to make creating your own hardware profitable.

Third, you're going to need to reach incompetent customers to make this profitable.

Yeah that isn't sufficient. Most people are going to get their email rejected or spam filtered from many sources. Having an IP that is not blacklisted is not sufficient to have it have enough reputation to be accepted by large providers.

There's an easy way to make Helm entirely uninteresting to spammers and that is for the server to limit the number of outbound emails that can be sent at a number that normal people would never exceed but that doesn't allow enough volume to be interesting to a spammer. There are much easier/cheaper alternatives for spammers today too.

Thanks for the question and your support!

It's interesting that Amazon is portrayed as the "massive corporate server" provider that stores your email "outside your home" on your homepage, yet you use AWS EC2 instances to pipe email traffic to/from Helm.

I understand that it's just an encrypted VPN connection, and are not actually storing email on the EC2 instances. But is there any way for your customers to ensure that? Can your customers shell into the Helm and/or EC2 instance(s)?

We will be making public the configuration for these instances as part of what we publish in open source. We haven't considered allowing customers remote access to the gateway but we will based on your suggestion. Thanks!

If you _do_ "consider it", I hope you discard it as an awful idea pretty much immediately.

I'd suggest allowing customers who're concerned to run their own instances with your code on it (perhaps a Docker image?) - but giving random customers shell access to gateways you're responsible for (at least to Amazon) - would be insane...

Signal/WhisperSystems are doing some interesting work on how to prove the code running on their servers is identical to their published and auditable code - might be worth checking that out (for a post MVP roadmap idea).

Yeah, there'a a lot of confusion about "the cloud". Consumer SaaS (Gmail, Facebook, Alexa, etc.) is "evil" and privacy-invading but B2B IaaS (EC2, GCP, etc.) is not bad. Often these services come from the same companies so we can't simply claim that Amazon or Google are wholly good or evil when it comes to privacy.

I believe you will be more successful if you establish two things, first your own ASIN with a couple of class C IPV4 blocks and an IPV6 block.

Then you create an infrastructure for relaying the mail from your boxes to the Internet. Then you work with the various spam agencies to create both a way to respond to spam complaints and to detect and throttle or cut off spam senders. You'll find that spammers will offer to pay you a premium to "look the other way" but don't take it, many good companies died going down that road. Without the spammers it will be harder to make your numbers but concentrate on keeping your efficiency high and ultimately you will be better off.

You don't talk a lot about data protection on site for things like disk failure. What do you do in that regard to keep people from losing all of their mail if the disk goes tits up?

and to detect and throttle or cut off spam senders

Isn't throttling outbound email count/day from the start the only real solution?

In my experience you can't know what is 'normal' until you have seen it, so fixing the rate at the start would be unduly onerous.

People have lives which can sometimes look spammy, like you are made the coordinator of the school potluck and suddenly you're sending email to 150 parents asking them to volunteer to bring food. But after that event you go back to your regular rate.

Because this isn't a "PC" in the sense that it is more difficult to be overtaken by a virus and start sending spam without your knowledge, and as a mail service provider you know that email originating from the device has to come from a specific domain, you have a lot more tools to detect that someone is being a bad actor or not.

No, the way the big boys get around this is outbound email filtering. Limits yes, session and connection heuristics, reading your outbound email to see if you're selling dick pills.

> hey jawns, great question. I'm Giri Sreenivas, co-founder and CEO of Helm. To answer your question, ISPs block port 25 and email service providers typically reject emails coming from residential IP blocks.

> To build a plug and play solution, we knew that our server could not require listening for inbound connections on a residential internet connection. So we set about looking into how we could route traffic to and from a home server but we needed to do this in a way that prevented us from being able to spy on traffic. We investigated solutions like sshuttle and eventually settled on the combination of a simple iptables configuration combined with a VPN connection. Helm establishes an outbound VPN connection to a dedicated EC2 instance with an iptables configuration that routes packets to and from the connected Helm server. The EC2 instance also has a static IP address associated with it.

> It's important to stop here and explain that the only way this architecture is viable while adhering to our design tenet of knowing as little about our customers as possible is because of the Let's Encrypt project. Every Helm server has a unique domain associated with it and trusted certificates for that domain are fetched from Let's Encrypt. We strive to ensure that all inbound and outbound traffic routed through the EC2 instance is using TLS with these certificates from Let's Encrypt. This way, our EC2 instance is effectively just an extra hop on the Internet.

> I hope that answers your question, let me know!

This doesn't seme to address the question of whether this violates the ToS, regardless of whether this is technically feasible.

Gray area IMHO. Isn't Dropcam basically a server streaming video to my phone?

Ha, but that's "okay" because it's a big company. They never have problems with being used for spam - of course I'm being sarcastic.

I'm pretty sure that video feed is actually sent to your phone through a Dropcam cloud server. Your Dropcam is a client which connects to the cloud server, as is your phone.

Arguably that's the exact same architecture Helm are describing. The EC2 instance your Helm box VPNs into is (arguably) "the server" - in that it's the "public service" endpoint. A good lawyer could eloquently argue that the Helm device on your residential internet connection is no more "running as a server" than the Dropbox or Google Drive apps on your laptop or phone...

(Note: I've been doing a similar home-rolled version of this, where I have a mail server under my couch that opens a reverse ssh tunnel and forwards port 25 and 465 from whichever transient EC2/Azure/DigitalOcean/Hertzner/CloudAtCost VPS Ive got listed as my MX records. A little bit of Route53 API automation, some Ansible to set up the ssh tunnelling, and some "canary" gmail and live.com email addresses to check outbound deliverability when I go to switch VPSs... It's been running pretty much untouched since 2014 or so...)

If I host a Quake 3 server I’m in violation of the TOS.

It can be argued that using a browser or any other client violates that language by sending responses to requests.

The ISP won’t do squat if it’s encrypted; they have no idea who is being connected to and what’s being sent.

A TOS is as broad and vague as they can make it. A good lawyer could undermine such language if it went to court.

Ultimately the TOS exists so the provider can discontinue your service, so it doesn't have to be super tight legally.

We don't believe it does. ISPs will only see encrypted traffic (a VPN tunnel to the gateway) so it's unclear how they will figure it's associated with a server.

Whether or not ISPs can figure it out is completely orthogonal to whether or not it's allowed.

For the most advanced of users, would you opensource the server so that we can host our own gateway servers instead of relying on Helm? That would solve the "what happens if Helm goes down" question and "how do we trust Helm" question.

Still wouldn't solve "how do I know my emails won't end up in spam" but at least we're getting closer (to at least what I would want to pay for.)

wouldn't that be postfix and a caldav server?

Also the fact that we can't run servers from our home connections is ripe for a challenge if we ever get net neutrality protections back.

The 2015 Order said this, "A person engaged in the provision of broadband Internet access service, insofar as such person is so engaged, shall not block lawful content, applications, services, or nonharmful devices, subject to reasonable network management."

I would argue that banning personal email servers or personal servers at all is not reasonable network management (e.g. a nest thermometer or a smart microwave or an Alexa/Siri thing is a server), and if we're looking to explore home appliances that decentralize the web, we need to ensure that broadband providers' policies don't block them. Google Fiber screwed this up too.

doesn't matter if we are contractually allowed to run them or not. The dynamic IPs of consumer ISPs are all blacklisted by the spam blockers. So, you could receive mail, but no-one would ever receive yours.

But you pay for this EC2 instance, and all traffic flows through it.

Honest question: What stops a malicious employee on your end sshing to this server and dumping plaintext messages from memory? What stops a court from ordering you to do that?

Even if you disable remote access, what stops someone from adding a new LaunchConfiguation that enables it silently on the next instance rotation in spite of whatever configuration is in place today?

At the end of the day it seems like you -can- spy on the traffic just as easily as you could if you were running the smtp services on an ec2 instance directly.

Given that, what is the value proposition here?

(Or if I am totally wrong, by all means call me out accordingly)

Well the E-Mail server still runs locally. While they could intercept traffic on their VPN endpoint, the traffic should be encrypted (TLS). However, I am not to sure if all e-mail servers speak TLS to each other.

Well if it does VPN first, then initiates the SMTP connection with TLS on the local smtp server all the way to the RX mail server, then this works out fine.

A lot of mail servers don't support this though, so it would be on the client to also be able to ensure it will not relay mail except to TLS endpoints verified by a well known CA.

In my experience this is rarely the case, but if it is and Helm is willing to tell end users "sorry I can't safely send mail to this endpoint" then I could see some value to this approach.

Yes - we initiate a VPN connection first to the gateway, then inbound/outbound connections are over TLS. Over 92% of email traffic is over TLS and we will be exposing an option in the future where customers can require it or reject emails.

In all honesty, couldn't this be a question to Amazon for anything that ever runs on any EC2 instance? What makes you not distrust any cloud vendor when they manage every bit of your information including keys.

Although, if they have to route stuff through EC2 instances anyway, I could just start jedieaston's homegrown EC2 email service, let customers pay me $100 a year to get them a domain name and spin up a email server in a EC2 instance and give them a login, then they don't have to have a box at their house which is liable to ISP and power outages. And you get the same amount of privacy that you get with this box (so about as much as you trust Amazon).

I -do- distrust all cloud vendors. My keys all reside in physically controlled HSMs.

Because traffic to and from the server will be over TLS, there won't be plaintext messages in memory on the gateway. We specifically designed to avoid the problems you have outlined.

Since most email's not encrypted, how is having each Helm user's email hop through your server any better for them in terms of privacy than just hosting their email on a remote mail provider in the first place?

You could still record every incoming and outgoing email as it goes through your server, couldn't you?

I really don't see the advantage of Helm.

Most (gmail claims ~90%) email is encrypted with opportunistic TLS in transit and can't be passively monitored.

https://transparencyreport.google.com/safer-email/overview?h...

Further, "the provider could intercept email" and "the provider stores all email" are very very different.

What's going on there with the "Support for encryption in transit" section?

Overwhelmingly at 0% in most regions which they say are:

> domains in terms of volume of email to and from Gmail, in alphabetical order

Something doesn't add up there.

The table does not include the "green" domains by default, you have to select it.

This is transport encryption, not actual email encryption.

Outgoing email from EC2 nodes isn't trusted by recipient domains either. That's why AWS wants you to use SES instead.

Amazon invests in ensuring their Elastic IPs are not on blacklists. Less than 2% of IPs we get through AWS are ever on a blacklist and when they are, we cycle through until we get one that isn't.

I'm not sure how they can invest in that.

I've seen amazon ec2 ips being used for ddos attacks and I blacklist the entire range for many of my websites/projects. The vast majority of visitors we get from there are abusive.

Thank you for this very informative reply. (I hope you'll put this information in a more prominent place on your site.)

I found the answer by clicking Technology from the top menu, and the explanation is the very first paragraph to be read.

“To a unique gateway” with a static IP is hardly an actual answer. Under whose AWS account is this server provisioned? Who has root?

It was enough to answer user jawns' question.

if your company folds and the cloud service goes offline, does that render my helm useless? that wouldn't make me feel like i "own [my] email".

Two things: we will run the service in perpetuity as long as there are subscribers and we will be open sourcing what is required to run the service on your own.

I thought that sending email from EC2 instance was not allowed and only option was using their SES service.

I'm pretty sure it's allowed, it's just that you'll get caught in every spam filter. I don't know if they've considered that problem, or they're just hoping that using the same IPs long-term will fix the problem in reputation-based blacklists.

You can request email sending support when you set up reverse DNS.

I guess email can be sent from the residential connection but recieved through EC2?

You can send email from an EC2 instance but good luck getting anyone to accept it. A lot of email providers block EC2 wholesale, or if they do accept it you are going to have to have a long standing reputation.

What do you mean by blocking "wholesale"? I started hosting an email server in EC2 and the worst I've had is my emails going to a spam folder if that person hasn't received an email from my address yet (and never after they've marked me as not being spam). That happened surprisingly rarely and didn't feel like much worse than I would get by just sending people email from gmail with an address they don't know. I don't think things are as bas as you make them out to be.

edit: I guess I should be clear that I have an elastic IP (which is free) and setup reverse DNS and DKIM and SPF, but I think those are fairly standard now a days (I don't know honestly I've only run an email server for a few months).

It depends on whom you send to. Like ironically I sent an abuse complaint to Verizon for a spammer and got rejected because they blocked all of Digital Ocean's IP space. Yahoo was particularly difficult too sending me a response saying they won't take my mail immediately on a single email to my brother. I had to go through a lot to get that fixed. Again this was due to being on cloud provider IP space.

GMail is more reasonable with perhaps being spam filtered but never blocked outright. I have also been blocked by government labs and academic institutions. I also have complied with RDNS, DKIM, SPF and got a top score on mx toolbox. Now that I have been up for a while I have had less issues besides with the ones that block cloud provider spaces.

Microsoft is the toughest one. Fortunately, I don't have friends that use outlook.com or w/e

Does your email get blocked if it is sent to a business on O365?

haven't tried in a while. It can, they dont' follow all the rules and do a little more blocking in the interest of their users or based off a ML spam detection or something.

Some things that should be delivered are not. I'd have to dig back into this to see what the exact issue is.

unfortunately DO does not seem to be as active as Amazon in maintaining the reputation of their IPs.

Deliverability is a moving target, so you can’t really rely on your delivery metrics from last week.

Why? The SPAM Scoring Industry is a kind of monoculture that will impede or cancel delivery of messages. Most big email providers outsource SPAM scoring to a third party like Symantec or Brightworks. And when your email is found offensive by Brightworks with one of their clients, you’re banned on all of their clients. Part of how SPAM scorers gather intel us by hooking into the CFL system so they get a notification when a user hits the junk button to delete an email. Another way is by checking for mass duplication in your emails. Another way is tracking open rates via the email provider’s UI sending metrics to the scorer’s APIs.

And then there’s the CIDR block lists they coordinate and distribute as a courtesy for their customers. So you might be able to deliver to @gmail.com and then @yahoo.com won’t even take a connection from your server’s IP.

Why does all this happen? According to my friend who works for a midrange ISP on their email service, they have about 120PB to store real email. And the amount of SPAM they reject outright (never reaches any box, not even the junk box) is 93% of all their email traffic. They simply can’t afford to store all the SPAM.

This is very consistent with what our experience has been with Elastic IPs combined with email authentication and blacklist monitoring.

> I guess I should be clear that I have an elastic IP (which is free)

Free when in use, you pay for it when it is not in use.

This is not a god situation. So, EC2 are handy and flexible servers, but not really because some arbitrary internet services are banned, and in all of a sudden your whole setup depends on the cloud provider. Talk about tight coupling. We go backwards. This is really sad.

You can send email using EC2 instances, including mass commercial campaigns (aka "legal spam"...).

However, there are a few things to do, more specifically, you must contact AWS to get "clean" IP ranges (fresh IPs, never allocated to instances), sign some agreements and pay additional fees.

It's not exactly as simple as an API call to create an ec2 instance, but it's doable.

With random IPs, there is a good chance indeed that it is already burnt by a previous allocation.

> I thought that sending email from EC2 instance was not allowed and only option was using their SES service.

Amazon EC2 throttles traffic on port 25 of all EC2 instances by default, but you can request that this throttle be removed: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-p...

Doubt that AWS would allow anyone to send on behalf of 3 parties like Helm appears to do though.

That EC2 instance is going to need whitelisting and constant vigilance that you're not sending commercial email or they'll block that port too. What prevents me from deploying a few hundred Helms to send spam?

commercial means spam? I can't use this for my non-spam business? MyStartup.com? A few 100 helms would cost a lot (500*200 = 100k), no one will spend that much for spam farm, you'd do it on the cheap.

What VPN solution are you using? Wireguard or something else?

Do you have a plan to help make email security features easy to consume: SPF, DMARC, MTA-STS etc.?

Is there a way to opt out on the information that is collected?

I looked on your site but it does not show a logical architecture which might be useful for some of us.

We are using StrongSwan right now. We've taken a close look at WireGuard but have not yet completed our evaluation.

We automatically configure SPF, DKIM and DMARC for our customers. We are also investigating MTA-STS.

Device diagnostics are opt-in by default so they are not collected. Customer data like shipping and billing information is not opt-out unfortunately as we need to be able to process payments, ship the unit and track warranty coverage.

Appreciate the feedback on wanting more architectural details. This will be coming in a series of technical posts explaining how we designed and built the product. Stay tuned and thanks for your questions!

What else runs on that EC2 instance?

I would first try using port 587 and only afterwards route through Amazon or something.