Lamson's Bounce Detection Algorithm | Hacker News Reader

Lamson's Bounce Detection Algorithm

lamsonproject.org

23 pointstwampss19 comments

I don't check my spam folder. If it's in there, it's not delivered.

Email providers should (optionally) bounce spam, rather than sending it to a spam folder. This would eliminate the problem of false-positives, since senders would be notified. It would reduce traffic, since spammers would know the message didn't get through to any inbox, and so would focus elsewhere. Finally, it would eliminate the temptation of a spam folder for people who don't want spam, but peek 'just to make sure', and thereafter go on to support spammers.

Perhaps someone here is running an email provider, or working on such software?

Bounce it where? The return address is always fake and sometimes it's that of an innocent bystander.

Not if it's a false positive!

I'd 550 the spamming server sending it to me, they can take care of the bounce themselves.

The problem is that the overwhelming majority of spam has a false return address, and the technologies to verify that mail came from the server it claims aren't yet widely enough used to be reliable. The bounces create an extremely large amount of traffic for servers unrelated to the spammers.

Besides, many servers do behave in the way you specify. Before I had the sense to disable catch-all email addresses on a few domains I owned, they were all being flooded with bounced spam, much of which my spam filters didn't catch.

Blah. Everything's a mess.

The important part of what I said above is that the spam would not arrive in any inbox. The temptation (and ability) to look through spam is the current problem, and would be eliminated.

If some servers are already doing that, then they are doing the right thing - except, the message body of a spam should not be bounced. The extra traffic on the bounce is worth expending, since it is necessary to keep spam out of sight.

If they can't get their message through, they lose the incentive to send spam. That's all that matters.

On the mail servers I run messages marked as spam get a 550 Spam error at SMTP time. The vast majority of users prefer this, as they can't be blamed for missing an email because they didn't look in their 'spam' folder, and the sender knows pretty much immediately that their message hasn't got through (in the event of a false positive).

As I discussed in an earlier thread, our custom mailer uses Gmail on a custom domain for storage; fetchmail connects via IMAP and then pipes to procmail which pipes to a Python script.

Lamson looks like it can offer a lot of great features, but I'll have to run my own SMTP server and I'm worried about downtime. If I let Gmail store the messages, I don't need to worry (much) about messages going missing. If my server crashes, fetchmail will just pick up the missed messages in the next fetch, only a few minutes late.

For others who run mail servers: how do you handle downtime?

With redundancy, basically. But there's not much "downtime" with mail servers, it's not like web servers where they're always being updated with the latest code and the app server needs restarting. Any admin worth his salt should be able to guarantee very good uptime (barring HW/power/network failure) for mail under normal/medium load conditions; high load has its own challenges but that's a separate issue. Of course, real 100% uptime means multiple machines at multiple sites, cue standard cost/benefit analysis.

And it's also important to bear in mind that it's an SMTP server; unless it's actually your users trying to connect directly it will be other mail servers talking to it. If you're down for a while, they will queue at their end and retry periodically. You'd have to be down for days to lose mail, although your users might get warning notices, which might look unprofessional.

One thing you could do to get the benefit of having your own server, while keeping the gmail redundancy, is simply set gmail to forward all those messages to you. If you went down, they'd just build up until you were back up, and that way you could avoid users ever getting those embarrassing failure notices.

I think Lamson looks awesome, btw. I started an SMTP server in Ruby before but gave up in horror when I started having to actually read those hideous RFCs. Shaw certainly has a knack for getting the dirty work done. A real mail server is a horrible job to write. He should be given a govt grant or something to do this shit.

(edited to sound less like an asshole, unfortunately not before OP had replied .. sorry >_<)

It's not much of a trick and it certainly isn't very complex.

fetchmailrc:

    poll pop.gmail.com with
    proto pop3
    port 995
    user "user@mydomain.com"
    password "pa$$w0rd"
    no rewrite
    ssl
    mda "procmail -f %F"

procmailrc:

    LOGFILE=$HOME/log/procmail.log

    # Process the mail and let the application do its own logging.
    :0
    | python2.5 $HOME/procmail.py

    # Alert the admins of any failures and keep a copy of the failed message.
    :0 c
    | python2.5 $HOME/alertfailure.py

Then start fetchmail to run every 10 seconds or so:

    fetchmail -d 10

Oops, sorry, I actually removed that bit since I thought I sounded like a douchebag. Serves me right for using "edit" to change my "drafts".

Thanks for the explanation, that's simpler than I imagined. But you had said IMAP, which is way more fiddly than pop3, since you'd be holding open connections and all that stuff, so I was envisaging having to run long-running processes, etc, which can be a headache in itself. Is gmail really happy with you polling a POP server every 10 seconds all day every day?

I meant pop. Oops myself :-)

Honestly, I haven't the slightest clue if Gmail minds the polling. I assumed that by the time we had enough mail traffic for it to be a problem, we'd be prepared to migrate to something like Lamson. Release early/often and all that jazz :-)

Hm, interesting. I hadn't even thought to use pop like that because I just assumed the admins or their automated scripts would get annoyed and block me.

Yeah, good approach. Just do the simplest thing that could possibly work; upgrade if and when you need to. I should perhaps tattoo that somewhere since it never seems to sink in : /

Besides what sho said about SMTP uptime not being that important, Lamson can read messages from a Maildir queue. You could use Gmail -> fetchmail -> Lamson.

http://lamsonproject.org/docs/deferred_processing_to_queues....

Aaahh cool. I didn't think about putting fetchmail in between. I was assuming I'd have to do Gmail forwarding to Lamson. We'll probably migrate to Lamson eventually to make good use of things like this neat bounce handling. Thanks!

From the page that says what the Lamson project is about (emphasis mine):

"I’m always looking for people doing interesting things with e-mail. Even if you’re a spammer, or someone trying to destroy the spammers, I want to hear from you" (Zed Shaw, maintainer of the Lamson project).

He's made the point before that any advance in email technology will be primarily used for spam.

http://www.theregister.co.uk/2009/07/03/lamson/print.html

"I totally think Lamson will be used more for spam than for anything really useful. My hope with Lamson is that it acts like a modern sewage system for the internet. Everyone's gotta poop. You can't escape it, so why not try to do it in a clean fashion that improves the general health of everyone involved?"

He's also said that he hopes it will mean spammers end up better at targeting so that there will be less shit flowing overall.

Makes sense. The huge amount of backscatter ain't helping anyone

Reply on news.ycombinator.com