Hacker News Reader

Comment by zcdziura | Hacker News Reader

The problem is that spam was/is so bad that extreme measures were taken to curb it. There are all kinds of invisible forces that you abutt that can be difficult to figure out, such as IP blacklists and the like. And even if you set everything up properly and host your email with a responsible host, Microsoft will still mark your mail as spam.

I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot, relaying all outbound mail through SMTP2Go (their free tier more than meets my needs). I have all of the necessary DNS entries set to mark my mail as legit, and I sign all outgoing mail using strong 2048-bit RSA keys. Thus far, I'm able to send mail and not have it marked as spam (at least to everyone that I've corresponded with thus far). It was a lot of work to get to that point, but not terrible.

View on HN

Prediction: Any distributed social media (like Mastodon) that gains mainstream popularity will share the same fate. Sure, you'll be able to host your own Mastodon instance, but 99% of people will be on the top 10 hosts and they won't peer with you.

I think the only way to make distributed social media practical is to have an extremely inexpensive turnkey self-hosting solution for the average person. A Chromecast-like device that they plug into their TV that backs up all their photos, plays music, and also hosts a Mastodon instance. Some kind of very friendly backup solution where you make an "emergency contacts" list, and the device encrypts all of your data and stores it on your emergency contacts' devices as a backup, and vice-versa.

cough XMPP federation cough

Not only did Facebook and GChat refuse to peer with little players, they refused to peer among the big players too. We could have had something like IRC for the masses, peered chat servers with bring-your-own-client. Instead, we waited decades for iMessage to get Android support which only happened long after everyone moved on to IG, Messenger, WeChat, etc.

Email is probably one of the last great open[ish] distributed systems we’ll ever see. There are just too many incentives to build walled gardens instead.

I'm not sure what the competitive edge could be to not wall garden. It's always going to be more expensive to try to work with those who don't work with you.

Going the self-host route, I'd still want a service of some sort so I didn't have to maintain it myself. Almost like an evergreen program that self-hosts my data and synchronizes and backups and transfers anything and everything.

Everything would be accessible outside of the program as local human readable or viewable files where possible. That'd be the best way to be non-walled garden.

> I'm not sure what the competitive edge could be to not wall garden.

I've had several email providers die since the 1980s. Each time it was a major disruption in my life. The last time, I coped by mostly dispensing with email whenever possible. Like most people I have a "good" email address for important things, that I check weekly, and a "garbage" email address I only bother to check when I have a need to.

Hosting my own mail server, not subject to some provider's ideas of filtering, or simply vanishing in the night, would make email more attractive. But the festering mess that SMTP email evolved into isn't just something you can set up in an evening. It's not even a hobby. It looks pretty much like at least a part-time job. Weighing the options, I don't really need email that badly.

> I'm not sure what the competitive edge could be to not wall garden.

If only one entity does it, as far as I can imagine it is only a marketing statement to appeal to a niche demographic - people who care about it from an ideological standpoint.

If more than one entity does it, it could lower the bar for critical mass. Instead of having to get enough people on your platform to start benefiting from the network effect, you only have to get enough people on your platform and platforms that you have bidirectional integration with.

It's actually fascinating to see the re-emergence of "vines" but as parts of other apps. With the explosion of Tiktok, every platform decided to make their own version - YouTube Shorts, Facebook/Instagram Reels, etc.

Really, a format should be created (e.g. file.[short|vine], etc.) that could be then edited by any editor and viewed by any viewer, and all that you'd need to do to copy a YT short to a facebook reel is to copy the file itself to each platform.

It's literally the same exact concept over and over again, just wall gardened instead. So much wasted development time doing the exact same thing.

Does this give credence to cryptoeconomics?

Exactly. GChat supported federation at the start, but they removed it later due to spam and other challenges [1].

[1] https://news.ycombinator.com/item?id=11795658

matrix.org has entered the chat

XMPP doesn't even support E2E encryption out of the box. It's as outmoded as IRC.

Lots of XMPP clients support E2EE/OMEMO out of the box now. If you think XMPP is outmoded you should try it again some time. I recommend Conversations on Android.

The problem with email is that identity and authentication are an afterthought. Don't forget that (in theory) it is possible to get any email server to relay a message for you. Newer protocols do not have these kind of problems.

>Don't forget that (in theory) it is possible to get any email server to relay a message for you.

That would be an open relay. That is simply not something that mail servers do anymore. If one was to deliberately set up an open relay, one would find that their email server was blacklisted pretty much immediately.

but then they just start a new relay.

I don't think so, I believe open relays are virtually extinct. People rarely run MTAs those days, and default configurations are quite protective. And if someone still manages to mess it up, they're gonna get famous with all the RBLs in days if not hours.

I self-host my mail for over 17 years. Most of the spam I'm observing those days comes from hacked/broken websites (sometimes it's probably some stolen SMTP credentials, sometimes sent from the server directly). Legit domain name, SPF and even DKIM present, looks totally legit in this regard - only stopped by RBLs and content filtering.

indeed, my ISP only recently closed their open relay for all customers

I remember back in the day having to change your SMTP settings whenever you travelled to whatever the ISP was where you were staying. then you could finally send email from your @homeisp.example email

If it was only open to their customers, that’s a closed relay, and typical for an ISP.

Open relays were a thing in the early 90's. I remember a friend of mine relaying email through 20 different servers, bang-path style. Any open relays today would immediately be used for spam, so they just don't exist, at least not for very long.

What was the original intent of open relays? Why allow emails without authentication?

> What was the original intent of open relays? Why allow emails without authentication?

Store and forward.

Do remember that email was THE great federated protocol.

The goal of a mail server was to get your email "at least one hop" closer to your destination. And that wasn't an easy task.

Servers came online and went offline. Users logged in and out. Connections came up and went down. IP wasn't the only transit. DNS? Oh, the hosts file? Even higher things--thing DECnet and Janet.

Email was barely functional most days. Your best bet if you weren't an Internet God and weren't able to write your own super complicated sendmail.cf was to know a sysadmin at a node who had an Internet God and ask him if you could forward emails that you couldn't handle to their server.

Email would be so amazing were it not for the spam problem. In the early days you’d just send a mail to your computer and your address was yourlogin@yourdomain and mail just ended up on your machine in a folder. Relays were like p2p networks. It was actually beautiful in its simplicity and in a perfect world with everyone being good actors could have been incredible.

And it was, back when any hint of "commercial use" could get your machine booted from the mail routes and usenet. After Cantor and Siegel, it was every spammer for himself.

Open relays were offered in the spirit of cooperation that was characteristic of the early internet.

Unfortunately, greedy people soon jumped in to take advantage of this generosity, resulting in a tragedy of the commons.

John Gilmore used to run an open relay, and I used to get spam from it. He was really stubborn about promoting the freedom of the spammers over the peace and quiet of the poor recipients. He eventually got shut down, still complaining.

http://www.toad.com/gnu/verio-censorship.html

> Why allow emails without authentication?

Because the people on the 'net were generally not bad actors back in the day, so we would you need to lock things down?

Hmm, now the internet is a dark forest. Leave anything open/visible, and all the bad actors immediately swamp you.

On Mastodon, I believe it's currently somewhat backwards from this. The largest instances are filled with Japanese anime porn, and the smaller instances end up blacklisting them.

Anecdotally, this has happened every time I've set up any kind of social media instance / discussion forum / BBS (back in the day!) / whatever. It immediately gets consumed by people who use it to host porn, and then all the intended users leave.

Have you considered creating a discussion platform where people can't post images, URLs / things that would be URLs if you added a URL scheme to them, ASCII-armored baseN-encoded anything, etc? For 99% of the discussion you want on the platform, text is all you need. For spammers and people who want to host porn, text alone is useless.

Plenty of spammers rely on text, though. A good chunk of my spam folder is text only.

Likely text containing URLs, though, which your email client likely helpfully auto-formats into links. Which is why spammers would bother. They need you to go visit something to make money.

There are types of bulk unsolicited email/web-comments which involve text without links, but they aren't spam per se; they're (the initial bait emails for) scams.

(And the usual solution to scam postings on forums, is to prevent people from sharing any off-forum contact details, except maybe via forum private message. No Nigerian prince is going to run their whole scam through the forum; they want/need to convert you ASAP into having a non-intermediated conversation with them. So they won't bother with any forum where they can't stuff their email address / Telegram handle / etc into each post.)

I've been itching for an experiment; might be fun to make a modernized BBS system for people to deploy.

If you're not spontaneously flooded with hentai, how do you even know your internet is working?

Japanese anime porn is not something that would trigger a banwave. A few posts from Trump, however...

Well, one of them is rotting the moral fabric of the country, and the other is just some hand drawn people having sex.

I was reading these tweets today confirming that this is already happening in Mastodon: https://twitter.com/Gravecat/status/1518598015396818944

This thread matches my experience with Mastodon and Diaspora*. It's fine if you are happy to live on individual instances and pretend that other instances do not exist, but they are not so great if you want a global audience. In this sense, they are more like the random disjoint online forums of the early 2000s, and not so much like the large monolithic social networks that people have come to expect.

Sounds like discord without voice and with easier linking. It does seem like forum approaches are becoming more common. I've heard that groups were the only part of Facebook with a lot of activity, but I'm not on that platform.

If anything, their story is more likely than not showing that the centralization is not going to happen. If the users of the instances were the ones doing the segregation (due to some tribal/cultural divide), then you'd end up with a small number of highly polarized instances.

But if this is only a fight between admins, the intuition is that we would end up with the big instances constantly losing users to smaller ones (created by those breaking away from the bad admins) who would then federate among themselves.

Even a magic dongle isn’t going to work. People don’t want to buy things, let alone sysadmin their own television.

I’d love a world where data was truly distributed and federated, but unfortunately, the barrier of entry is too high. Because of this people will start hosting nodes for people. Which isn’t necessarily a bad thing, but network effects will take over, and we’ll be back where we started.

Look at git. It’s distributed in all the right ways, but almost everyone uses github.

The web is decentralized, but the same few websites dominate to the point that people — even people on this very site — think that you can’t post a video except to YouTube.

Disagree. If the authentication mechanism are available from the get-go, it could work.

Unless you can reliably tie a user to their real-life identity, authentication isn't useful in this case. If a spambot tries to peer with my instance, it's not super helpful to know that their accounts will always be the same spambot and not a different one.

Identities could be signed by a centralized authority, which would have the same desired effect as centralized hosting without the drawbacks.

A person is not required. A reputation is required.

In a digital world, with no financial penalties, it's easy to build reputation with 'spurious' transactions and exhaust that reputation for one "Large Evil Event" and rinse and repeat.

I noticed a lot of German sites don't peer with anyone who has the exact same rules (as in almost literally the same) as them. I was surprised to see such kind of box-thinking in a protocol that's been designed to be as open as possible.

I assume this is unavoidable. The only solution are protocols where the network is owned and stored in the data (cryptographically) rather than in the servers. Then the servers apply censorship and rules over the data, but you can still rebuild any conversation chain as long as you connect to enough servers that don't censor it instead of requiring 1 server to keep all the network relationships.

This also allows authors to seamlessly switch servers without losing audience or at least being able to recreate it very easily.

That's another problem: Moving your account to a different server in the fediverse. Which is indeed not possible currently.

Perhaps some kind of blockchain would be a solution? (No, I'm not trying to appeal to tech investors, I actually think it might offer just the solution here :P )

nostr (https://github.com/fiatjaf/nostr) seems to be a minimal possible solution. It doesn't seem to be much in use though, so I guess once that happens a few issues will come up.

I hope matrix can avoid this fate. The federation can of course be limited, but currently it seems like most users are federated. And they have plans to combat spam etc. https://element.io/blog/moderation-needs-a-radical-change/

> Any distributed social media that gains mainstream popularity will share the same fate.

The experience behind this predated peer-to-peer electronic cash and related developments. You may be right, and it may still be too soon. But problems can be solved.

> But problems can be solved.

Looking at the 30 years and millions of dollars poured into making email work, the evidence seems to be against this

I don't agree.

There is no technical solution for people being assholes.

Well OK there is - turn off computer or server :)

One idea I've had is what if the protocol were designed in a way that a server can't be scaled too much, thus forcing lots of small servers to federate instead of having single entities running a large server with tens of thousands of users.

Arguably, your prediction is even a feature, not a bug.

The right to peer implies the right to not peer.

Agreed. I suspect most users have been tricked into thinking they want massive, global social media platforms.

(1) People are turning their noses up at Mastodon because all of Twitter isn't already there and because you'll be cut off from instances that aren't federated with yours.

(2) People are worried about "all of Twitter" becoming more people than they would like. There are communities they'd rather be cut off from and words they'd rather not read.

It's not a bug, it's a feature. Unfortunately, very influential companies that have figured out how to game our attention have tricked users into thinking they want something they don't.

> The right to peer implies the right to not peer.

No? Even if you don't want to force smaller instances to peer (which generally makes sense) you can apply more strict requirements to huge instances that contain a significant portion of the population.

> I think the only way to make distributed social media practical

The only technical way maybe. You could always legislate that large enough networks MUST peer.

Just because at one time the majority of users are federated, a market/threat force can enter any time that would drive users to centralised solutions.

If it's so easy to self host, surely attackers will host thousand of those instances and spam you.

PoW has been the best solution so far.

Can you give me an example of a PoW-based social network or chat application which has been more successful than conventional alternatives like Mastodon?

I am hardware stupid, but I have thought about this exact solution for so long. I hope someone figures this problem out!

General observation: no matter how distributed you make a system or protocol, it eventually becomes centralized.

Yup. Spam is the root problem. With an enormous amount of complexity between that and the mail admin's day to day experience.

I hosted my own mail for more than 20 years. A couple years back I just got tired of trying to solve deliverability puzzles, plus the fears that deliverability issues generate. (E.g., "Did that potential employer get my email about the job?") Especially since some of the puzzles are not solvable, like why GMail does what it does. I even had friends at Google, and I still couldn't find out why GMail occasionally didn't like my server. And arguably, that's the right choice for them, as the more spammers know about how they work, the worse it is for Google staff and GMail users.

For me, switching to Fastmail hosting was a big win. It's not like I'm out of technical challenges to solve, but I get to apply that to things where the upside is greater than, "The thing everybody expects to work still works."

the spam problem advantages google, as your own story illustrates, so it's unlikely they'd really want to help solve deliverability/spam issues systemicly. making personal email hosting more difficult means they have a chance to capture your email data streams via gmail. whether you switch there or not, it creates a pressure for most to aggregate on gmail, which means they can see most email exchanges.

For sure. Good spam filtering was one big reason for people to switch to GMail. And a lot of people who gave up hosting their own email have switched to Gmail as well. I'm sure this doesn't rise to the level of conspiracy, but there's little incentive for them to fix the broader problems.

My issue with fast mail et all is storage is so unnecessarily expensive. I have many gigs of email that I don’t want to lose, but I also don’t want to pay many tens of dollars/month to host it.

I use SES just for delivering emails while hosting the emails myself. Feels like an optimal solution.

> I have many gigs of email > don’t want to pay many tens of dollars/month

How many gigs? It appears that the 100GB plan is only $9/mo.

Whoa. That’s very different from the last time I checked, perhaps they changed either their plans or their structure. I saw it previously as additional storage I’d have to add. 100G would be enough! Wow thank you!

Is it? I pay $5 for 50gb, which is many years of email (including all the spam/newsletters, which I really don’t need to keep)

If you have iCloud you have the option of using a custom domain. I think I only pay a few bucks a month.

For various reasons I’d rather have an indie provider. I’m currently on Google and trying to decentralize my data.

Ok, if the amount of data is a problem, another option is to just download your mbox from Google takeout, treat it as an archive, and use something like notmuch to search your old mails. You can then store and backup the mbox anywhere you want.

In addition to the iCloud, I have a free ProtonMail account, which I use sparingly, and anything large or important that comes in there I move to local backup and delete. 99% of the messages I get can just be deleted. If you set up pop3 you can auto delete from the server. That’s the old school solution, but it depends on your use case. The pop3 option doesn’t work well with multiple devices.

Maybe self-host an IMAP server, and periodically archive from FM to your box/VM?

> And even if you set everything up properly and host your email with a responsible host, Microsoft will still mark your mail as spam.

I did some experiments back when I ran my own mail. Sending from my mail server to my Microsoft account it not only marked everything as spam, it continued marking everything as spam after I marked a bunch of them as not spam.

After that, I tried also answering several of them and composing several new mails to send to my non-Microsoft email to see if Microsoft's spam system was smart enough to figure out that if I'm actively corresponding with someone their incoming mail should not be marked as spam. It was not smart enough.

Then I tried whitelisting. Nope, still spam.

Microsoft had marked an email from a professor from an vt.edu domain email address as spam causing me to miss an interview for a PhD funding.

Microsoft is especially notorious for flagging legit emails as spam if they are not from one of the regular providers.

Flagging if you're lucky, they outright 550 refused my mail until I joined their sender program and applied to have my domain unblocked. Then they proceeded to gaslight me claiming my mail was never blocked even after I forwarded their own error messages and IDs back to them.

You got a 550? Lucky! When I worked at a non-profit that hosted our own email server, we had many instances where Microsoft would /dev/null our newsletter e-mails. Their servers would give a 250 indicating acceptance, but the e-mail was nowhere to be found (and yes, we checked the spam folder).

yea unfortunately I have seen those as well. It is ridiculous at times.

That's nothing compared to the joy of dealing with legit emails that are flagged as high confidence phishing.

Regular provider == (Microsoft 365 || an Exchange Server)

Honest question- why can't people sue for this?

On what grounds would they sue for? Email is not the post; there is no legal right to receive one or to have one routed.

If one wants such legal protections, there is the post.

(Now, should there be such a right? That's an interesting question. But a world in which one exists would raise the bar to starting one's own email server even higher).

> On what grounds would they sue for?

Negligent interference.

https://en.m.wikipedia.org/wiki/Tortious_interference

Possibly, but it would be a hell of an uphill battle. There was no contract in place for the email provider to negligently interfere with. And the email provider's operation was perfectly regular and within the bounds of the standards of that service (which offers no delivery guarantees).

You can't sue if a product doesn't work as intended and results in harm?

It depends on the circumstances. In some cases, when guarantees are made and those guarantees are broken, you can sue civilly to be made whole (in a context like this where there was no bodily harm, merely an opportunity missed).

It's real unlikely any such guarantees were made. To do so would be extremely foolish for several reasons (the false-positive rate of spam identification is known and emails can fail to deliver because of an error at either end of the transaction).

You can't sue someone (and win) for "this person did something that I don't like". You only have a case if you have a contract with them that lays out specific duties, or if they are otherwise a fiduciary of some form. Unless you signed a contract with microsoft for them to deliver your mail, they have no obligation to do so.

A legal protection would mostly entail disabling of spam filters.

Email (SMTP) has no delivery guarantees. It's basically "best effort."

If you want guaranteed delivery with proof and tracability, send a registered letter at the post office, FedEx, etc.

unfortunately, access to the Internet is not well defined enough for this, and you basically have no right to a connection or any guaranteed privileges if you have a connection, which sucks.

No guarantees clause in the terms of service you accepted to use their product.

I was using free email, I don’t know if MS has any obligation

Former VT employee: It's not just Microsoft. Virginia Tech has a problem with their stuff getting flagged as spam.

I can't find it right now but a few years ago there was a post here on HN by a grad school admission officer who recommended to never use Gmail for applications, no matter what school you apply to. Apparently, it's anything but uncommon for emails from @*.edu to end up in spam.

Yes to the OP, you most definitely can host your own email fully.

Many of us do it. If you have any interest in the topic, either due to the fun of managing the servers and learning something along the way or due to the moral high ground of supporting decentralization above proprietary walled gardens, do it!

Ignore the naysayers, if you're interested you can do it.

Will some emails very occasionally end up in the spam folder of a recipient? I mean, yes, but that is true of everything. You can end up in spam folder sending from Microsoft Office mail to gmail or vice versa. Heck, every now and then an email from my manager will end up in my spam folder in gmail even though he's emailing me from gmail to gmail, both of us in the same corporate gsuite account! So on average, once you set everything up correctly, your deliverability will be as good as gmail to gmail, which is to say not 100% perfect but no worse than any other solution. And you'll be in control of your email infrastructure and address. No longer will google/microsoft/apple/yahoo be able to cut you off all your accounts on the whim an AI gone bad.

The parent post mentions a useful safety valve to know about if you're worried about deliverability and want to take baby steps to get there. You can always, either selectively or wholesale, use a commercial relay for outbound mail from your email server. Some have free tiers that are plenty for personal/family use.

Personally I don't use any third party relay, I deliver to everywhere from my own infrastructure. No issues.

Me too, and this is my experience as well. In the rare event that I find out that someone isn’t getting my emails, I tell them that they should complain to their provider or use a different one. I’m no longer willing to jump through hoops so that hotmail delivers my email.

> There are all kinds of invisible forces that you abutt that can be difficult to figure out

This was my main experience, and all I did was try to set up the ability to simply send emails to myself (gmail) (and no-one else). Things like: this script crashed, or btrfs scrub finished + scrub results, and similar.

The first thing I tried was just setting up a VM with postfix running on it locally with my residential ISP. I don't even remember what the error was for this scenario, but it was just totally dead in the water. Absolutely zero mail delivery. I think I eventually figured out it's because google defers to spamhaus, and spamhaus says residential IPs = hard no.

That next thing I tried, and what I ended up doing, was writing a docker container that just runs an SSH port forward to jump from my local network to a digitalocean host, which is where another docker container runs postfix. I had done this bit once before, and I tried to just set up DKIM (since DKIM was, to my reading, basically bulletproof - why bother with SPF when you have real cryptographic identity assurance?). This led to weird error messages from google about my IP having a super low reputation. This was something I'd been worried about so I spent a bit of time trying to cycle my IP. But I eventually figured out it was just a bad error message and setting up SPF suddenly made my emails start delivering.

My main ongoing issue is that I had to add all my sending addresses (things my internalhostnamehere@myrealdomain.com) to my contacts in gmail, otherwise there was like a 50% chance they'd just go to spam. I've been running this setup for about a year and it's still a coin toss whether emails will come through fine, or if they'll say "this would've gone to spam but it's in your contacts". When that happens, I check the DKIM and SPF status in "original message" in gmail, and gmail itself says they both passed.

Absurd tbh.

For my "not self-hosted but better than letting google own my digital identity" solution, since I use apple icloud+ or whatever it's called, I set up the SPF stuff to let me send+receive email from my custom domain, so while icloud could still scan my mail, at least if I get banned, I still own the actual domain and could move somewhere else.

Even if one setups everything by the book (SPF, DKIM, DNS.) etc. No one at @outlook.com will receive email, based on my experience. Thus, it does not work well if email is important for business-to-business use.

Outlook and Gmail are basically having opaque rules who can receive email and there is no process to get “whitelisted” on these receivers.

I had exactly this problem too. I elaborated below.

If you keep an eye on your logs, when your emails are being blackholed (it accepts them but it does not deliver them!) it does provide a link in one of the 550 status messages, where you can get yourself unblocked. I've elaborated here: https://news.ycombinator.com/item?id=31185297

However this only works temporarily, after a month you're back in the doghouse. Only senders which send a large volume of legit traffic are allowed. It's ridiculous but sadly true.

Edit: I found the message in my old emails:

---

550 SC-001 (BAY004-MCxxx) Unfortunately, messages from XXX.XXX.XXX.XXX weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors.

---

In that link the "SC-001" code also refers to that reputation thing. This was the same at outlook.com / hotmail.com and live.com . It did not, however, affect corporate customers using Office 365 / Exchange 365. Only customers of MS' consumer offerings.

My "internet service provider" was a legit colocation service and nothing funny was going on in their network by the way. Microsoft was the only party that had issues with my server. All known blocklists had no issues with it. It was just MS being difficult and making up their own rules.

Anyway going to that link there is a form somewhere to temporarily unblock it. Give it a try.. Perhaps you can create an account at live.com yourself and send a daily test email or something... I thought of doing this but eventually I got so frustrated I gave up on it.

> Only senders which send a large volume of legit traffic are allowed. It's ridiculous but sadly true.

That’s the thing I’ll never understand: why would one have to spam email to be considered not spam?

I wonder at which point it is legitimate to go full conspiracy theory, and suspect they’re just trying to block the little players so they can keep their relative monopoly. Maybe they don’t do it on purpose, but the way their anti-spam measures make life real hard for the little ones sure looks convenient.

If you get a 550 then it did NOT accept the mail. What grandparent is talking about would be the server replying with 250 OK and then neither delivering the mail nor sending a delivery failure notification.

One potential 1%-of-the-complexity answer to the problem of personal notification (which I presume was what email-to-self was solving) is to set up a Telegram bot. I recently really wanted realtime notifications on my phone (package tracking) and realized that all the top-level "send notifications to phone" type services are either mass push notification shops ($$$$) or bundled offerings ($$$) that were entirely overkill for my purposes.

There are two ways to run a bot on Telegram, either by running the bot client directly (meh, interesting but extra setup) or by using Telegram's bot hosting system that works over HTTPS. It's the second approach that takes 3 minutes (!) to get to an MVP state for notifications.

- You walk through a flow with a specific account (@Botfather) on Telegram to create a new bot account, which gives you an API key

- Find the new bot using the search function then open a conversation with it and (after sending /start) send a junk message

- Call `curl "https://api.telegram.org/bot$APIKEY/getUpdates"` and fish out the "chat"->"id" value from the JSON representation of the message you just sent to obtain your user ID

- Call something like `curl "https://api.telegram.org/bot$APIKEY/sendMessage" -X POST -H 'Content-Type: application/json' -d '{"chat_id":"1234567890","text":"boop"}'` (set chat_id to your account id) to send a new message - yup, it's literally this simple to send messages

- Go into Telegram's settings and add the bot as a notification exception (assuming you have notifications universally turned off by default)

- If you also set the full-screen popup to "when off" Telegram will (even when your device is locked) show an instant notification containing the sent text

- Because this is a conversation, the message history will be preserved unless you explicitly delete the messages (which you can do on a per-message basis)

- The Telegram bot API supports both polling and push-based I/O, where you can periodically poll /getUpdates or have Telegram call a webhook you configure. IMHO the way easier approach is just running the bot client locally at that point, *but*, for just sending out one-way notifications where replies don't matter, the default polling setting (no webhooks) is ideal as the bot server will delete un-acknowledged messages after IIRC 24 hours or so - so you don't have to worry about queue quotas or whatever, you can just ignore the whole receive side and it just works

Obviously the caveat is that this is 1% of the complexity and equally 1% of the... provenance, for want of a better way to put it. But in terms of "I need realtime notifications now" I am yet to find a better system. It worked perfectly.

This is the answer. Blocked emails happen for random reasons and fixing them is a black art that involves talking to ISPs and stuff. It's really too much for an average person to handle.

At work we've had issues with email delivery due to things like outdated IP block lists at some random ISP four hops away, only impacting deliverability when mail gets routed through that part of the web.

I have an email address with "spam" in the name (this is through gmail) and lately I've had all kinds of problems with emails to it disappearing - I've had to call several places and have them change my email because I can't log in and the reset emails don't ever show up... but changing to myfirst.mylast works fine.

I've run into this with both Sam's Club and Speedway Rewards.

Only thing I can think of is that some outbound mail service they're using is dropping them, or some relay in the middle is dropping them... I can see where the word "spam" would be a keyword you might use, but I've had this email address for 15 years now and it's only been a problem in the last few years.

I have this part:

> I host my own email server with Vultr on an OpenBSD VM using OpenSMTPD and Dovecot

But with outgoing mail being relayed internally to dkimproxy which signs it before being relayed back to OpenSMTPD for delivery to the other email server.

I had to set up SPF and DKIM DNS records, and one time I had to request that my IP be removed from the Abusix blacklist. Other than that, it's pretty rare for my emails to be marked as spam. Outlook 365 seems to do it much more often than Gmail though.

That's very interesting. I never thought to relay mail internally to dkimproxy. I'll have to give that a shot. I like the idea of hosting the entire solution myself and not relying on any 3rd party solutions, but relaying through SMTP2Go was the only thing that I tried that actually solved the problem. Perhaps this will offer a good solution! Thanks!

I also use the dkimproxy package, but there's now a third-party OpenSMTPd module that can sign messages in-line.[1] I've always found dkimproxy setup a little confusing compared to a built-in/in-line solution. I might try to switch to the module during the OpenBSD 7.1 upgrade process.

[1] I think this is the one I had I mind, though I didn't realize it was already in ports: https://cvsweb.openbsd.org/ports/mail/opensmtpd-filters/dkim...

If you run a mailing list you generally have to worry about ARC (re-signing 'chain of custody') in addition to DKIM:

* https://en.wikipedia.org/wiki/Authenticated_Received_Chain

I've found ARC to fiddle some to get going than ARC.

> relaying all outbound mail through SMTP2Go

So it's not an entirely self-hosted solution, is it?

No, but it's quite difficult to have email reliably and consistently delivered to Gmail and other major email providers without sending it via a relay. The relay provider is in the business of maintaining IP addresses with good reputations that aren't blocked by spam lists etc. If you can find and keep a reputable IP address, then you're fine, but it's usually easier to pay someone who does that for a living—you have no guarantee that the IP address assigned to you by Digital Ocean or whoever wasn't used for spamming at some point.

Digital Ocean has an extremely poor reputation over a long period to the point where their droplets are blocked on mass in many places now [1]

Even my local ISP refuses mail from them.

[1]: https://discourse.mailinabox.email/t/digital-ocean-ips-being...

Really sorry, I don't normally nit pick spelling and grammar, but it's "en masse" rather than "on mass".

This is also why I went with Vultr as my server host. They block port 25 by default and make customers file a support ticket with them to unblock that port. They also require your account be active for at least a month and be using their service in good standing during that time. Wasn't an instant process, but was simple enough to accomplish in the end.

Yeah, lots of places just straight up block entire IP ranges, such as anywhere you can get a VM for cheap/free, or residential IP ranges, etc.

> The problem is that spam was/is so bad that extreme measures were taken to curb it.

Man, and there's such an easy solution, too - just use Hashcash[1] (invented in 1997) and 90%+ of spam disappears overnight (if not more, depending on how high you set the difficulty).

Well, ok, "easy" in the sense that We Have An Algorithm For This - it'd still be hard to get email clients/servers to agree on a protocol...

[1] https://en.wikipedia.org/wiki/Hashcash

Ah yes, time to break out this old classic: https://craphound.com/spamsolutions.txt

  ( ) Requires immediate total cooperation from everybody at once
  ( ) Unpopularity of weird new taxes
  ( ) Public reluctance to accept weird new forms of money
  ( ) Huge existing software investment in SMTP
  ( ) Sending email should be free

It's fortunate that none of those are "checked" in the ASCII art, because none of them actually apply.

> Requires immediate total cooperation from everybody at once

False. As a silver lining to the Google/Microsoft email oligopoly, those providers could announce that anyone wanting to send email to those services will have to implement this protocol, and it could be done in less than a year.

> Unpopularity of weird new taxes

Irrelevant. No taxes involved - there's no money here, and users won't care if their mail takes an extra few seconds to send, because they don't expect email to be low latency anyway

> Public reluctance to accept weird new forms of money

Irrelevant - no new money involved.

> Huge existing software investment in SMTP

Irrelevant - a small number of server software are used by the majority of users. Also, see earlier point about oligopoly.

> Sending email should be free

Bad idea, and irrelevant, because it still would be.

I suggest you put thought into copypasta before putting it into a comment.

If there's a compromised machine it will be the victims paying the cost in energy bills for spammer's nefariously installed malware to send garbage.

If there's a compromised machine, the scammer can drain the victim's bank accounts and cost them far more than an electricity bill, and/or mine cryptocurrency directly. Regardless, their spam-sending rate will still be significantly decreased.

So, this argument is completely invalid.

> If there's a compromised machine, the scammer can drain the victim's bank accounts […]

Not if the machine is a server and was gotten into via (e.g.) a bug in a web app. I don't know about you, but I don't keep my bank account information on the LAMP systems I sysadmin.

That part of the post was specifically about consumer devices.

You missed the rest:

> and/or mine cryptocurrency directly. Regardless, their spam-sending rate will still be significantly decreased

Your argument remains invalid.

Is there such a service that will tell me the reputation of an email domain, i.e. whether mail originating at that domain would be likely to be treated as definitely spam or not? (I don't really care about "no reputation"; I want to know if a domain has known bad reputation.)

I feel like, if there was such a service, it would be pretty useful to use it to prevent account registrations on other services, from users whose email addresses have domains with bad reputations. After all, they'd very likely just be registering with the intent of using the service to send or post spam in some way.

multirbl.valli.org

Contains blacklists on the domain level, also on the ip block and AS level.

Possibly interesting... but these are rules about outgoing SMTP servers (MSAs), yes? How much of a relation does the outgoing SMTP server for a domain have to the canonical set of receiving SMTP servers (MTAs) for the domain held in the domain's DNS MX record? These can certainly be one and the same server; but it's not a requirement. So how often are they in practice? Especially for people actively trying to evade these sorts of RBLs?

> The problem is that spam was/is so bad that extreme measures were taken to curb it.

The problem with spam is that there's no real legal recourse for spam. If it's in your own country then maybe. But outside of your country? Well the easiest thing to do is to IP block and the next best thing to do (when IP block isn't an option) is to use some sort of "smart detection" to put spam into a special box labeled "spam". There's no deterrence and literally no criminal prosecution for spam.

Also, one should subscribe to the mailop mailing list, which serves as a Distant Early Warning line for email deliverability issues (ie. like NANOG for netops issues).

https://www.mailop.org/best-practices/

zddziura, I want to say a big thank you, thank you, thank you for pointing me to SMTP2Go. I have been trying to get my DMARC and DKIM email woes solved for months, but couldn't get it figured out. When I read your post I signed up with SMTP2Go at the free tier and I had a 100% Mailgenius score in less than an hour after I set it up. So awesome! No more big yellow warning boxes in Gmail when receiving mail from my own domain! Yea!!!

Spam got significantly worse but this is also an chance to curb the federalization of mail by large companies. Of course they would like you to use a Microsoft or Google account to send mail.

Any chance you'll provide a detailed write-up of your experience with tips and whatnot?

Reply on news.ycombinator.com