Initially, it seemed like a DoS to us too, but it was not. This was confirmed by upstream provider metrics. No major traffic spikes. It was a combination of non-malicious things. More info later, some of us need sleep.
742 karma · joined June 24, 2010
I feel like there is an element of "Body Doubling" here... a strategy used by those with ADD/ADHD. I recently looked in to this when curious about my own observation that I work longer and with better focus when working in close proximity of someone else.
This thread is making me feel a lot less crazy.
Unbelievable. Thank you for sharing your experience!
I'm a big fan of HN and YC in general, we host of other YC alum, and I have taken a few things through YC Startup School. During this incident, I spoke to YC personally when they called this morning.
See more complete job description: https://www.ziprecruiter.com/jobs/m5-hosting-inc-2672ed89/sr...
Can I answer any questions you have about us?
A basic 1Gbps commit on a 10Gbps port in a data center might cost you from $0.50/Mbps (something like Cogent) to maybe $1.50/Mbps (let's say Level 3), other providers could be $4+/Mbps. By the time you factor in all of the above overhead costs, the true cost of the bandwidth is much much higher on a per Mbps basis.
Don't forget to significantly over-build your stuff, or you might get knocked off-line for anomalies or DoS attacks.
Admittedly, the scale of Google, AWS, Azure makes the cost per Mbps much much lower, but when as others have pointed out, AWS, Google, Azure don't need to charge less than they do.
In the years since the rolling blackouts, I have been colocated in data centers that have made the decision to use special pricing available from the power utility, available when you can have a preemptable load… during peak hours of peak season the power company can tell you to get off the grid with short notice. The data center operators did OK when they did this for most years. They got a lower price on power all year, in exchange for running their generators more. I’d say that the typical year, they ran the generators for several hours per day for 7 to 10 days in the Summer months. The last year they dd this, they ran a much higher number of days than expected… maybe 15 - 18 days from noon to 7pm. I understand they had to stop doing this because of the pollution of the generators or the permits required for such heavy usage... but that info is not first hand. It could have been many other reasons such as neighbors complaining about the noise in the business parks (2MW diesel generators are deafeningly loud), costs related to running the generators for so many hours, etc.
To operate like this, you have to be on your game for maintenance of the generators and checklists and training. You also have to have multiple contracts with fuel delivery trucks, just incase your outage lasts for a while. These data centers were all under 5MW in size each. We never lost critical load during a power supplier outage. I hope I have illustrated that a reliable power supply is not strictly required to run a reliable data center or service that is dependent upon a reliable data center.
[1] https://en.wikipedia.org/wiki/California_electricity_crisis
Yes, it may take time. But you don't get many NSLs, so you do it. You do it to protect yourself (liability of disclosing info without a legal order) and your customer/user who is the subject of the letter. Every time.
I thought NSLs were supposedly non-contestible, broad and were for communication detail. These don't seem to be any if that.
The requests we have received have been from a variety of organizations (but signed by a magistrate) ranging from local law enforcement to three letter acronyms and one entity that is neither. While the requests don't say why the order is being issued, we usually receive a call from the agent/detective beforehand and dialog ensues in which they explain what's going on.
While many companies will just give the info, we scrutinize the request and ask the agent/detective politely and apologetically that we can help, but only if they acquire a court order. We have caught not-legitimate requests before, so we verify the request is legit before responding. We have never been asked for content of communications. If Google is not doing the same thing... oof. Just as a matter of process I assume they do. I recall in the past some networks having right in their WHOIS info, how/where Law Enforcement can send FAX requests.