Sendgrid is down
support.sendgrid.com
support.sendgrid.com
DDOS of a mailing service that lots of websites rely on because a completely unrelated company decided to fire someone is not an occasion for lol and schadenfreude as some posters here would have it. As a method of justice it has more in common with a lynch mob than a court of law - this isn't going to get the guy's job back, and it certainly isn't going to teach anyone a lesson, apart from that the internet is fickle, and monumentally stupid. But I very much doubt the people behind this attack are interested in justice or truly care about the man who lost his job, they're just doing it for the lolz and are punishing the internet at large over a silly little dispute at a tech conference.
Congratulations to the mob, I guess; it has shown its power, if not any sense of discrimination or proportionality.
This should be a lesson for companies that hire highly confrontational people as their official community representatives.
Yes, in a perfect world there shouldn't be DDOS' and other attacks because of a tweet about an immature joke but it's not a perfect world and we should be very wary what personalities we hire to represent us.
As much as I don't like it - the "right" thing for sengrid would be to replace their 'developer evangelist' with someone who's less confrontational. Yes, it sucks. But as businesses we have to deal with reality.
You mean like when someone gets personally offended by a comment and instead of resolving it one-on-one turns to her Twitter account to shame and blame the "violators"?
Yeah. Except the person who instigated this set up the lynching in her capacity as a professional, i.e. on her "evangelist" twitter account. The kids DDOSing aren't representing companies or doing business
I'm really sad these guys lost their jobs because of these remarks, I don't think the way she handled that is the way we need to move forward.
[1] in this case, the guy who was called out for the jokes, posted in a comment on here that he was fired
I don't think so. It's incredibly difficult to tell the difference, given that a DDOS is a huge spike in traffic.
On the flip side, I'm happy that this story has been up-voted and that it attracted some attention because I was able to know that Sendgrid is down (never received any mail notification and I don't follow them on Twitter).
As much as I don't support the methods, seems like this actually sends a message (right or wrong). So it perfectly works from attacker's perspective.
I'm pretty sure by now every critical person in Sendgrid know the details of what happened, why happened and people involved. Not because of the initial twitter discussions but because this actually costs them money directly.
heroku addons:add mailgun:basic
heroku addons:docs mailgun:basic
vim config/initializers/mail.rb
git add config/initializers/mail.rb
git commit -m "Switching to back-up mail provider."
git push heroku master
(For us in particular, SendGrid only represents a small amount of the email we send and it's mostly internal emails. The problem for us was that the ActionMailer emails aren't sent in a background process, so this caused a couple request timeouts.)
I have also noticed that false positives from other startups in my own spam folder are often sent through Sendgrid (though I suppose that's not dispositive since it could just be that more startups use Sendgrid).
edit: when I made this comment I thought this was a random service failure that would last couple of minutes. More than 1 hour later, I don't think it's that funny anymore as I'm being affected as well.
P.S: I still think Sendgrid are awesome and fortunately they'll listen to what we have to say and next outage will be handled differently.
Now the mob is mad at Sendgrid.
Will you also feel schadenfreude when they hack her bank account?
Second, she wasn't "standing up" for herself because nobody wronged her. Overhearing a third-party say something about a dongle that you construe in a sexual light isn't being wronged, and the world doesn't require you to "stand up" to it.
This is not because she's a woman. Even if a man had done such a "stupid" thing (according to DDOS-ers), the result would have been the same.
I don't dispute for a second that the tech community (and indeed HN) has a huge problem with sexism that manifests itself in very ugly ways, but let's stick with the facts here.
I will definitely feel schadenfreude if her bank account gets hacked. After all, she did rid a person of their income by her actions.
1. The guy who got fired had an overzealous manager who fired with insufficient cause.
2. The guy who got fired had a history that we don't know, but the manager does, and this was the final straw.
Both theories are possible. I personally have known a higher portion of guys fitting #2 than managers who would enable #1. Therefore conditional probability suggests to me that he was fired for more than just this incident. If so then his firing would not be her fault.
(Even in #1 the firing was not her fault - it was the manager's.)
But in this case, it's not a simple binary case of was/was not "her fault". It's at least partly her fault, regardless of the accuracy of scenario #1 or #2. There is a chain of causality here, and it all starts with Adria's tweet/blog post.
I'd also argue that your two theories represent a false dichotomy, and sussing out your conclusion based on the idea that there really are only two possible explanations (throw in some personal anecdotes for good measure!) is, well... lazy.
I'd also argue that your two theories represent a false dichotomy...
Actually they don't. Are we agreed that this event is insufficient cause for a firing? If so, then if this event was the real reason for the firing, then the manager fired for insufficient cause, which is my #1. If not, then there is more to the story, which is my #2. Those two possibilities are therefore logically complete.
However they are not mutually exclusive. There might have been more to the story, and yet the manager still fired for insufficient cause.
That said, what's going to happen now? The guy who got fired has just become a cause. If he's got any skills at all, he'll get another job. I'm confident of it.
Adria has become radioactive. She hasn't been fired, but if her employer keeps getting attacked because of outrage over this, that's a possibility. If she does get fired, she's going to be radioactive for a while. Her line of work requires her to be public about where she is. And no sane company wants to be included in the outrage aimed in her general direction.
2 posts.
Happens all the time, always has.
Since many services rely on SMTP providers like Sendgrid, they should have a way to notify customers when their server go down and transactional notifications may be disrupted.
I shouldn't be notified by someone who know we're using Sendgrid and happen to read HN.
Also seems like best practice would be to have SPF/etc. entries in place for multiple ESPs, even if you routinely just use one, and be able to switch, for just this reason.
What it means is that basically you should be using 2 providers at the same time and send 50%/50% on each provider in order to keep your IP warms.
While I think big services should account for such scenario, I think 95% of Sendgrid customers are using them to avoid setting up that kind of redundancy.
Main question is should I create the same redundancy for my hosting provider, DNS services, CDN etc...
DNS is interesting, especially if you do anything location-based. I currently use just CloudFlare, but am not convinced they have enough internal redundancy on DNS, so investigating using Dyn, or self hosting again, or Route53, or some combination. It gets a more complex if you want to use multiple providers doing anything beyond simple DNS though (the DNS protocol itself is totally fine for this, but most of the config management is provider-specific, and using normal dns zone transfers/notifies doesn't really work in this model)
Update: For some reason I couldn't reply to your response kstrauser so here is my reply: That works if and when their is trust. First you have trust the buyer isn't going to abuse the system and wreck the IP [reputation] and you have to trust the buyer isn't giving you stolen info. Obviously things one can work through but again, it involves trust. Also the provider has to have warmed up IPs to give out. Which, having warmed up IPs would actually be a great valued add upsell for those that need them!
Update 2: Thank you FfejL and symfoniq for explaining it better.
I'm not saying this to advertise but to offer a suggestion: call someone and ask for help. This isn't an unusual need at all and any reputable provider will be quick to help.
Short version: you don't have to "warm up an IP" if you do a little advance homework.
If Google (or any other ISP) suddenly sees an big spike in email from @example.com on an IP that @example.com hasn't used previously, Google is much more likely to mark those emails as Spam.
So senders need to 'warm up' an IP by sending a small amount of email first, usually for a few days at least.
So if you're a SendGrid customer expecting to send 2,000,000 emails today, you can't just switch to a new ESP and send those same emails. Spam rates will go through the roof.
I would expect them to send emails to customers in the event of serious issues though, expecting your customers to learn about problems through Twitter isn't that much better than learning about them from HN.
Was what I was replying to, not insinuating the fact that SendGrid's website going down is a "PR Slide"
Letting users set "I must ack notifications of outages or keep pinging me" would make sense, too.
I'd probably consider mobile push notifications with confirmation and/or voice or SMS as well.
Basically like Nagios, but for third-party components.
Am I understanding this right - transactional emails for my company may be interrupted because of some random personal argument between two people?
https://news.ycombinator.com/item?id=5400134
EDIT: It seems that her personal website (adriarichards.com) is down too.
* require you to use user:pass instead of keys
* no historical service status page
* poor notifications about downtime
* unclear explanation of uptime
I am guilty of using SES myself, but it's sad to see email becoming increasingly centralized.
While I'm sure a lot of SendGrid's customers don't want (or don't know how) to configure a mail server, there are other customers who know that delivering email isn't as simple as installing Postfix. The rise of centralized email services is the inevitable byproduct.
As services go, its lock-in seems very minimal.