Inside the seedy underworld of spammers and phishers
blog.mailgun.com
blog.mailgun.com
Another way to fight spammers, would be to quietly shut off sending for their account, while still providing simulated email data to their dashboard, reporting successful sends, opens etc... That way, they would think they are still sending out spam and it would take them a while to realize that they had been cut off, slowing the cycle of them doubling their efforts.
However to make it happen our system needs to have 0 false positives and we are not here yet. If your system made a mistake and you just disabled the account, angry customer will appear on the chat support in 5 minutes and the problem is solved. On the other hand if you pretend to be sending, you can loose all the customer's emails and that'd be nightmare.
And this is obviously something you can't discuss, but I wonder if you guys "seed" various sites with false emails as "markers". It would be a very cheap way to detect indiscriminate scrapers and bulk spammers.
I assume the same thing would happen with email; spammers would use a few of their own email addresses on the list to make sure the emails are actually being delivered. It would be a pain for a while for sure.
The legal and ethical issues surrounding this strategy is something to consider, though.
Since when has a spammers return on investment been low?
Since when have spammers only used hijacked "legitimate" business domains instead of just using some wildcard email domain setup?
Its not enough that he posts his strategies online to make it easier for his adversaries to learn from, but this guy doesn't even sound like he grasps the fundamentals of what is supposed to be his profession?
To be on the same page in this conversation we are programmatic email service for developers, not for end users. Customers can create their own virtual email server on our page and start sending in a couple of seconds. This concept is pretty similar to cloud servers.
> Since when is a botnet a collection of free email accounts?
In our terms botnet can be a mix of a free and paid Mailgun accounts. Botnets can include anything from 2-3 to dozens and hundreds of accounts created at different time and using different billing plans.
> Since when has a spammers return on investment been low?
We are talking about Mailgun service - the time they need to invest in building some solution on top of Mailgun that pays back is just not worth it. Actually I'm surprised why they even bother sending this type of spam through Mailgun. Let's say they were able to send 100K of emails via us (what is pretty hard nowadays btw), in the best case their click rates would be floating around some fractions of a percent.
http://www.sitepoint.com/spam-roi-profit-on-1-in-125m-respon...
So they wont' get even a couple of clicks from that.
On the other hand, phishing attacks are very dangerous and this is our biggest threat - we've noticed that they get very high quality lists with 0 bounces, so it might be real bank users and build pages for every atack.
> Since when have spammers only used hijacked "legitimate" business domains instead of just using some wildcard email domain setup?
Wildcard MX records are about receiving, I'm talking about subdomains on a free webhosting services (bulk subdomain creation), what is a serious threat.
> Its not enough that he posts his strategies online to make it easier for his adversaries to learn from, but this guy doesn't even sound like he grasps the fundamentals of > what is supposed to be his profession?
Botnets and targeted phishing attacks are not somewhat new - that's a common practice, it's not that I'm uncovering some unknown secret here.
My criticism is I've never heard of a botnet referred to as a group of accounts. To me, a botnet is a group of host computers that run some sort of proxy server (tens of thousands of hosts). I've never given thought to what someone would call a group of email accounts aimed at exploiting a service, but to me botnet seems like it would be specific to a network of computers, sometimes compromised, sometimes not, running a type of proxy or automated software.
> We are talking about Mailgun service - the time they need to invest in building some solution on top of Mailgun that pays back is just not worth it.
The problem is when it comes to a service like yours, if it really is that hard to bulk mail, then the guys using your service aren't the guys getting a 1 in 125m response rate.
Anyone sending that kind of volume would assume their messages were going to a spam folder, and sending larger volumes to compensate for it.
Someone going through the hoops you set in place, are doing it because your service gets inbox. This means they can send bulk email to higher quality lists, and their response rate will be significantly higher than 1 in 125m, more like 1% to 2% response rates.
In this case, the people actually sending mail through your service probably don't even bother making those accounts themselves. They likely find people who specialize in circumventing your security measures, and pay a premium of $x to $xx per 1,000 accounts.
> On the other hand, phishing attacks are very dangerous and this is our biggest threat - we've noticed that they get very high quality lists with 0 bounces, so it might be real bank users and build pages for every atack.
This too, but don't forget about simply cracking passwords for the accounts. Simple math. Take the top 100 most used passwords, assume your users are just as naive as most the internet, and you have x% users you can assume will be compromised at some point in the future.
So for old school spammers even if they got lucky and got 1% click rate, they'd 1K clicks in their best day in our service. So I'm mostly considering them as people looking for potential holes in the service.
The people coming with stolen credit cards who want to steal more are the biggest threat as they are most harmful - they hunt for our ips and domain reputation, so they take time and try pretty hard to break through our filters.
> This too, but don't forget about simply cracking passwords for the accounts. Simple math. Take the top > 100 most used passwords, assume your users are just as naive as most the internet, and you have x% > users you can assume will be compromised at some point in the future.
Yep, and we watch every account in the system for changes in behavior, but that's happened only once or twice in the last 2 months - so it's not a biggest problem right now.
Turns out nobody other than me cares about the zoom changing on rotate, and everybody wants pinch to zoom.
As an ESP, isn't that pretty much the game you chose to play, both as the cat and the mouse?
Would be more curious to know about how effective they have observed this to be, or maybe more about what they learned a long the way, profiles are always interesting, how many false positives, customer complaints(& support time) etc. Maybe a future post?
Yep, we were trying not to disclose too much information on how we catch them, however I agree that how we fight them deserves a separate post.
Some things to share:
* Naive approaches (hey, just plug in spam filter) don't work in most cases as spammers tune and create the new content specifically for our service
* Feedback (complaints) from customers is a great signal, but at the point you start receiving the complaints it may be too late.
* Bounce-based metrics (invalid addresses) are a great signal.
* There's no silver bullet as we've found, you have to collect as many signals as you can
* Rules based systems don't work as the rules change every day, you have to plug in some learning in place.
* Domain blacklists are also not very effective - as they use hijacked domains, or services providing free sub-domains to avoid blacklists.
* Ip blacklists are not very effective as well, as a lot of people are now using cloud services sharing the same NAtted ip.
* A lot of customers don't really realize they are spammers - "Hey, we've paid money for this mailing list, it's all fair"
It may not have been something which you wanted to do, but I think it is a really interesting problem, and I bet it has been rewarding for both business, and in a pure engineering sense.
In some ways, I think about what it must have been like to create a fake identity in a less connected age, and I wonder at how it will continue to evolve.
I recall some Doctorow novel in which spam and its increasing sophistication was almost an escalating arms race between our ability to distinguish authentic interactions versus those that were staged or generated / general sock puppetry.
I am curious about additional signals and information, I would presume in addition to fingerprinting and collecting as much information about each of their implicit touch points, did you find yourselves increasingly relying on more traditional manifestations of identity/reputation, etc.
edit: Or I wonder about a discount for new sign ups with a one time facebook scan & score type mechanism :D
Thanks again for sharing more information, good food for thought!
Talking about traditional ip, domain and complaints reputation - it helps a lot to identify and block ignorant senders using some questionable techniques for getting their recipient lists, but it's pretty useless for fighting phishers - you need to act immediately and automatically, and reputation takes time to aggregate.
I've been dealing with some non-email spam recently, and after reading this I count myself lucky -- most of the stuff I see is SEO related and they tend to come from distinct IP ranges and can be surfaced with some simple rules. I'm sure as time goes on, they will become more wily.
That said, in my view, that's the entire point of your service. Every language has an SMTP library; the hard part - the thing I'm paying you for - is the constant cat and mouse.
It'd be great if mailgun did a blog post dedicated to what their Razor does and how they built it.
Once it's clear we are trying not to offend anyone in case of some terrible mistake, but instead ask a couple of simple questions to quickly verify the identity or business. We've found that this is the only way to avoid confusion.
However in some cases it gets very tricky - they can control the domain or can build an website with some non-working signup forms and services. We proceed and launch investigation up to signing up for their service - usually sign up does not work :-)
http://www.spamhaus.org/faq/section/Generic%20Questions#103
However I think it's generally a great idea, I'd also love to collaborate with anyone to fight spam.
Probably not well enough to use in an automated system, but perhaps well enough to be used merely to flag things for careful/further inspection.
I think the best example of a US spammer was this guy attempting to promote his blog. (Or trying to warm up his account before sending spam) Each post was a rip off of an article from About.com. All the post dates were adjusted to appear that the site was online for months. His spelling and grammar were excellent.
He had several flaws... The domain was just registered a couple days ago. The "Corporate HQ" address on the site was a post office in New York. The billing address was a UPS store in Nevada. He refused to talk to me on the phone or provide a physical address that I could send a t-shirt to. :-P
I've been exposed to what bank fraud and phishing scams can be like, and the craft is really amazing.