Linode down in Newark data center | Hacker News Reader

Linode down in Newark data center

status.linode.com

34 pointshendler34 comments

"We have detected a large incoming DDoS attack to our Newark facility. We already have our administrators working to mitigate the attack. We will have service restored as soon as possible."

    [dan@palladium]# traceroute 66.175.213.170
     1  * * *
     2  ae12.dar02.sr01.wdc01.networklayer.com (208.43.118.150)  0.349 ms  0.351 ms                                                                                                                                     0.325 ms
     3  ae9.bbr02.eq01.wdc02.networklayer.com (173.192.18.188)  0.951 ms  0.966 ms  0.975 ms
     4  * * *
     5  * * *
     6  * * *
     ...

Where is that quote from?

A support ticket I had opened before there was anything on their status page. That was the response at 11:53 PM Newark time.

Same here, got this reply from support :

"Linode 1 hour ago

Hello there,

We have detected a large incoming DDoS attack to our Newark facility. We already have our administrators working to mitigate the attack. We will have service restored as soon as possible. Please refer to the following page for status updates:

http://status.linode.com

If you have any other questions please let me know.

Thank you.

Nick"

Interesting their reply says that, while their status page says it is due to a power outage.

Edit: Nevermind, j15e clarified.

New response from support :

"Based on the information we had at the time we believed it to be a DDoS attack. We are now aware that it was a power problem affecting our Newark data center. The backup system did not work and we are still looking for additional information from the data center staff.

We will be updating our status page with any further information.

Thank you."

Seems like it was caused by a power outage.

  1:09AM EDT (UTC -4): The Newark datacenter has suffered from a power outage. We are working on bringing Linodes back up as soon as possible.

As others have mentioned, we've seen reports of network and power outage, and now I've received a notice of hardware failure.

Just when I set up HA load-balancing, the whole data center outage takes it all down for 4 hours and counting.

    Hello,

    The host that your Linode resides on experienced hardware
    failure. As a result, drives are being transplanted to
    separate, known-working standby hardware which we keep
    available in case of events such as this.

    Again, we're very sorry for any inconvenience caused by
    this issue. Please feel free to update this ticket in the
    event that you have any additional questions or concerns.

Does Linode automatically check for, detect and replace faulty hardware? Or is this something you have to explicitly ask them to carry out?

Normally they just detect it and raise a ticket like this (which happened here).

They previously had a message that Linodes would be returned to their previous state. Mine was rebooted. Just FYI. Check your DBs and all that usual stuff.

There goes my impressive uptime...

Mine was rebooted 2hrs ago but still cant access it.

That happened to mine, too, though not for as long.

I bet they had some host filesystem corruption and they're dealing with it manually.

Thanks for the encouraging words ;) Still not up, 5+hrs outage so far... EDIT: it was up, but the network interface was messed up for some reason. Rebooted again and its ok now.

Nodes are now being reported as stopped, but can't be started yet.

"An issue affecting the physical hardware this Linode resides on has been detected.

We are working to resolve the issue as quickly as possible and will update you as soon as we have more information.

Your Linode will return to its previous state once the issue is resolved.

Thank you for your patience and understanding."

We have tens of servers across 4 Linode data centers. We get hit with both data-center-wide and single-server downtime pretty much on a weekly basis.

In the span of 3 months we had something like 25% of our servers go under "emergency maintenance" downtime. Just part of the ebb and flow of "the cloud", I guess.

Wow, that sounds pretty unlucky. Do certain data centers/servers have more downtime for you than others?

update: "The Newark datacenter has suffered from a power outage. "

so, no DDoS after all?

Is it just me or does it seem like there have been tons of DDoS attacks the past couple months.

It sure feels that way, but I don't know if they are just being better reported now. Services to mitigate DDOS have been around for ages, so I imagine the attacks have as well.

It doesn't help that every time a service gets hit and the result is a frontpage HN post, more people notice that DoS attacks work for gaining "attention".

My understanding is that many recent attacks have accompanied ransom demands.

DDoS attacks have been on the rise, I work at a hosting provider and last year we only saw somewhere around a thousand attacks, now we're seeing that monthly...

My server auto-rebooted and is back up, and has been up for 30 minutes as of writing (approx 1:00am EDT / UTC -4). Strange that mine is up and fine now, but they haven't updated the status page yet. Maybe mine is an isolated case?

Is your node location "Newark, NJ, USA"?

Mine are still very unreachable.

Yes, in Newark. I was also affected by this outage earlier. Looks like maybe I'm one of the luckier ones (relatively speaking).

I feel like this should be included in every press release post down-time.

"As issues unfold, information changes and speculations can lead to improper actions based on incorrect information."

So very well said.

Hanging out on #linode, it appears as if some people's servers are coming back up. This is the second day this week I've woken up to unpleasant computer news :-(

... and mine are back up.

Complete blackout for my nodes, can't reach them at all. Even Linode dashboard stats for my nodes are empty starting 11:51.

Almost 30 minutes now, I hope it get fix soon!

I have 12 websites there. zomg. Looks like they are alive now, after 2hrs outage. Did someone cut the cable for scrap metal?

I think this is the first time I've ever noticed my own website being down.

Some of mine back up, others not. Not how I wanted to end my Saturday night.

Evidently DC issues are fixed, since some of my servers are up again. Others still down due to (related?) hardware issue. Linode support proactively opened tickets about those that are still down, and upon friendly questioning, expressed a high level of confidence that they'll be back up once drives are migrated to stand-by hardware.

Reply on news.ycombinator.com