Linode Outage in Dallas, Fremont, Atlanta, Newark, and Toronto Data Centers | Hacker News Reader

Linode Outage in Dallas, Fremont, Atlanta, Newark, and Toronto Data Centers

status.linode.com

152 pointsboplicity103 comments

Arrgh! :( Just this week I needed to do some emergency rearranging of personal infrastructure, and moved one nameserver to Linode Fremont, while the other nameserver is in Linode Dallas. I was fully aware that I needed a more diverse long-term solution, but I figured "What are the chances that both Fremont and Dallas will go down in the next week or two?"

Sorry, folks. I hate that my dangerous flirting with Murphy's Law is hurting other people.

For whatever it's worth, since it was a DHCP problem, I was able to manually configure the static IPs and get the hosts back up. The last incoming connections logged before the outage were around ~ 6:24:00 am MDT (about one hour ago). I guess that's when my server needed to renew the DHCP lease and couldn't.

In almost 20 years with Linode, I've had nearly zero problems. Akamai buys them, and within months we have an outage. Not encouraging...

On the other hand, it's only hitting my personal website, and the customer I have on Linode seems unaffected, so that's nice at least.

To be fair, I know from personal experience Fremont has gone down hard for hours before.

Others have had different experiences.

https://news.ycombinator.com/item?id=14571234

I'm wondering how much of this is perception...

The number of reported issues coming through has exploded since Akamai. Over the last week alone, almost every day a new networking issue for London is in my inbox. However - none of the recent ones have actually affected any one of my Linodes, even though it's possible since it falls under the service and datacenter - perhaps the change we are seeing is more transparency?

I'm not saying no one is affected... as others in this thread clearly are, and I was affected by the datacenter wide outage in London earlier in the year - but it's possible that being more upfront in notifications of issues could change the perception of what before would have been considered a collection of isolated incidents.</speculation>

[edit]

I generally appreciate transparency, but when it comes to varying degrees of service disruption maybe there is a balance to be struck... right now it feels like the signal to noise ratio is low in terms of it's affect across all users even within an area.

Right now I am pretty much ignoring the Linode issues filling my inbox, if a serious one came through I'd probably miss it.

I’ve had a non trivial number of issues with them over the years. Enough that I moved all of my infrastructure to other (more expensive) providers.

That being said, it was probably value for money spent while it lasted.

Well, it appears to be fixed now (for my websites at least), so I'm not as worried about Akamai as I was a couple hours ago. If it's another decade before I have an outage, I'm satisfied. On the other hand, if this happens again any time soon, it will be time to start evaluating options, I guess.

I've also been with them nearly twenty years (!?!), and they definitely have had problems throughout that entire time. perhaps the difference was that you'd get caker@ and co on IRC explaining what was up.

I moved off Linode 5 years ago due to outages and issues. I don't have any experience post-akamai to compare but it was definitely never a reliable service for me in the past.

Which site ? I've been in Fremont, then London with basically zero problems (few scheduled migrations for when they were changing hardware)

I searched my emails and it appears it was the Fremont site. It was about 7 years ago I moved off Linode. Initially I went to EC2 but then I just replaced most of my VPS needs with other specific services. FWIW, EC2 was more reliable IME, but obviously the Linode UX was way better.

FYI: Linode says that most people can fix this by enabling "Network Helper." You'll then likely need to reboot your server. It worked for me!

https://www.linode.com/docs/guides/network-helper/#individua...

Thank you! this worked for me too. I was totally down, went in and enabled Auto Configure Networking per your link, and then rebooted.

Most of my Linodes are still accessible, but this has worked for the two that went down so far.

My Newark node was down for over six hours, this is how I fixed it. Uptime on the VPS had been over a year before this (don't remember why it was rebooted a year ago). Like others here said, I did not greet the news of Akamai buying them with joy, and things like this starting seem to bare my initial impressions out.

this.... replaces the use of DHCP with a static file rewriter on boot, OK, why is this recommended in general, should we expect DHCP to have ongoing issues that their "network helper" would not ?

For the vast majority of people, a Linode is just a virtual server that sits on a static IP like a fat toad.

This network helper just assigns that static IP instead of assuming DHCP will find it.

You could turn it off later I guess.

Yeah, we’ve only reallocated an IP address once (in the past week, as it happens), but that was an exception. Our servers are redeployable and backed up, but they’re not really ephemeral. So they just have their IP “forever”. I think the oldest virtual server we’ve had was deleted when it was almost 3 years old.

They were just bought by Akamai so unfortunately things like this are bound to happen if they are trying to integrate with Akamai the way they'd need to successfully keep growing. I'm not talking just software integration but process and people integration as well. We'll see I guess!

I don't think many users will be affected by dhcp outage. Most linodes have fixed IPs.

Mine don't seem to be affected

> Most linodes have fixed IPs

I would expect the opposite. Doesn't DHCP make more sense, for example, if the user creates a snapshot and restores it with a new IP? I don't use Lenode, so I can't judge, but I would expect DHCP to be default.

Default created Debian based linodes use a static IP. How they assign you the IP is they have an initial network boot script that injects that static IP into your config.

I learned this because I wanted to test image recovery of deleted linode. When I restored it, I got a different IP address from linode (the static IP assigned to me), and I couldn't connect to it. Using the web console I found the IP address was hard coded to the old one still.

Maybe this changed recently but I created a few Debian instances this year, both were configured with DHCP by default

Just checked my tiny Nanode 1 GB image (Fremont) running Debian and it's indeed running dhclient. Still have access, but I need the instance this weekend for a virtual event. Kind of sucky timing for a potential outage, Linode.

That being said, maybe check if you have `dhclient`, `dhcpcd`, `systemd-resolved`, or (less likely) `dnsmasq` or `bind` running (htop / ps aux).

If you do, your instance may fall over after its DHCP lease expires and it then presumably fails to acquire a new one and goes all 169.254.x.x on you.

The possible counterpoint / information-vacuum bit here is that I don't know what the DHCP lease time is, so the expiry may already have happened.

IOW, apparently this started 4h ago, and so the last leases were presumably issued 4h ago, and maybe the lease length is 1h, so everything's already as dead as it's going to get, so maybe if you're not already dead, you're fine.

For Linode Ubuntu 22.04 installations, at least, the default seems to be DHCP, even if the IP address is fixed. And I certainly wouldn't have thought anything about this, since even when using fixed IP addresses, I'm in the habit of using DHCP to provide a central source of IP address truth for my networks, to ease the pain of the inevitable renumbering.

> Most linodes have fixed IPs.

We run some Windows instances, which were DHCP out of laziness. <facepalm> [edit: It's a PITA to fire up a Windows instance on Linode. Once working, switching out of DHCP is near the bottom of the todo list.]

Most have DHCP-fixed IPs.

The IP lease never changes but it's assigned by DHCP - check your settings (mine was in /etc/netplan/01-netcfg.yaml

dhcp4: yes

I'm absolutely certain Akamai will fuck Linode up, same as Rackspace ruined Slicehost -- which is what sent me to Linode in the first place.

I'm waiting for a new Linode to pop up.

Out of curiosity what is a new competitor going to offer that the existing well established ones like DigitaoOcean, Vultr, etc arent?

I would much rather deal with a human-scale firm than a giant one. That alone would get me to switch, even at greater cost.

DigitalOcean doesn't have much Asian presence, Linode's backup doesn't support non default file system like zfs, DO / Linode don't have HDD block storage like Vultr does (which I think is a hidden gem you can't realize until you go add one as it's not mentioned in the pricing page) that can be a very nice option to throw backup data in at 40GB/$.

Vultr's CPU isn't as good as the other 2 by the look of sysbench, even for high frequency types.

Those 3 vendors only have 2 or 3 backup slots which can be kind of unreliable if you didn't notice data loss over a weekend and weekly backup can be too far apart.

Hetzner is almost perfect if they have a bit more location. Very cheap, very high performing, daily backup with plenty of slots but no HDD block storage.

Sometimes Amazon Lightsail can be a decent option as they have 7 daily backup slots, free bandwidth is comparable to the others and prices aren't bad except their CPU suck.

Small things but none are perfect.

> DO / Linode don't have HDD block storage

Linode have had block storage for ages, I use them for ZFS.

I meant "HDD" based block storage which should be far cheaper than SSD/NVMe based ones when you don't need performance.

I don't think that will happen with Linode (but I certainly could be wrong). When Rackspace bought Slicehost, people didn't really know what the future was going to look like. At this point, we've seen AWS/GCP/Azure and Akamai wants to have their own "run your apps with us" system.

Unlike when Rackspace bought Slicehost, Linode probably isn't even the leader in its niche. That's probably DigitalOcean. Akamai probably figured out which of the two it could buy and DigitalOcean probably would have cost $4+ billion while Linode only cost $900 million. When Rackspace bought Slicehost, there were really two alternatives: Linode and AWS. AWS came with a lot of complexity and Linode was small.

If you're looking for a new Linode to pop up, you don't have to wait. Vultr, DigitalOcean, and tons more already exist and have for years. It's not 2010 where you don't have loads of options in this space. Go with Hetzner or OVH Cloud and get really cheap prices (I've found Hetzner Cloud to be really nice). There are so many places that will give you a VPS today. Heck, AWS will give you a Linode-like experience with Lightsail, right?

That's why I think Akamai wants to use Linode as a way to build something that will let them start getting into the AWS/GCP/Azure space. It's somewhat where Linode was already heading, albeit slowly. Linode started offering hosted Kubernetes, object storage, block storage, hosted databases, and load balancers. How long before they might have added a queuing system, an app-runner so you don't even need to think about boxes/load balancers, etc.

For the low price of $900M, they could jump-start that process at Akamai. Rackspace thought that the future was their high-touch, high-cost, managed services. Akamai probably wants to use Linode to get into the AWS space and with a separate brand, they can do it without as much risk to their mainline business. There are too many alternatives for Akamai to be deluded enough to think they can just ruin Linode and not like that $900M on fire. That wasn't the case when Rackspace bought Slicehost.

Akamai looks at Cloudflare's valuation with envy (and some contempt because Akamai has the larger edge network). Definitely agree Linode was purchased as a growth opportunity.

Cloudflare's valuation is for sure because its free therefore everyone uses it(~20% of the internet).

If Akamai was also free and as accessible as Cloudflare, they would start to grow for sure.

> I'm waiting for a new Linode to pop up.

Eh, what? Vultr, DigitalOcean, Hetzner, OVHcloud already exist. The first 2 at least have feature parity I thought.

Switch to Pair - https://www.pair.com/ - they are older than Linode and have a great reputation. (PS: However, I am not sure if they have been acquired by another company now. The "old" Pair had a sterling reputation, and I can vouch for that as a former customer.)

I'm paying for a few small Linux VMs. Pair has nothing all price competitive.

They're also now part of a giant media company (!), so yeah, pass.

Look up "managed" hosting vs "unmanaged" hosting - that's one explanation why their services are costlier.

...but I don't _care_ why they're costlier, and also noted I don't really want to deal with them as an entity, so it doesn't matter.

Just pointing out that price comparison between two product or service needs to be done with similar yardsticks. A product or service being costlier doesn't automatically mean that it costs you more as it can save your business money elsewhere.

It doesn't really, though.

I'm happy with what I'm getting for the price I'm paying. I'm not interested in a bigger spend here at all.

So yes, it's completely fair to dismiss Pair as being too expensive based their prices alone.

Pair is absolutely not price-competitive. Pair's "Starter I" plan is $30/mo, for 1 vCPU, 2 GB RAM, 20 GB storage, and 500 GB transfer. Linode's most similar plan ("Linode 2 GB") is $10/mo, and includes 2 TB transfer.

Does look like they have hiked their prices. But it also depends on how you are making the comparison. For example, just a cursory look at both pricing shows me some major differences - Pair is offering "fully managed" VPS hosting which means the hosting providers employees will help you do most of the common hosting tasks (and provide better customer support) unlike "unmanaged" hosting where you have to do everything yourself with the tools provided. Is the Linode plan you cited also similar or for "unmanaged" hosting (which is what Linode was famous for)? Note that Linode (Akamia) also makes a distinction between "shared" vCPU vs "dedicated" vCPU and prices it differently. Don't know how Pair looks at that. The point is that the pricing variation can be justified by the services being offered. I admire and respect Akamai and also like Linode a lot. But I also recognize that Akamai is big B2B company offering enterprise services and small or independent developers will always be an after thought for them.

Serious question: why are you stanning for Pair? Do you work for them?

No, I don't. I am vocal about brands I admire or dislike. One of the ways I was taught to evaluate web hosting companies was to check how long they have been in the business (both the company and the management) as web hosting is an easy business to get into (even today, any Tom, Dick and Harry can start a web hosting business with next to nothing investment by becoming a reseller). Pair hosting is one of the oldest web hosting companies I know, and they have / had a reputation for high server uptimes (I still remember one of their employee had once posted in some forum on how one of their FreeBSD run server had an uptime of more than 2+ years - yeah, bragging about server uptimes was once a thing too) and great customer support. (I also have fond memories of Linode, Slicehost, Rackspace etc. before they all gave up and were gobbled up by someone, and web hosting became all this "cloud" bullshit.)

Tl;dr: No. Brand recognition / recall is a powerful marketing tool.

Some old businesses are still running because they've been able to stay relevant over the long term. And some old businesses are still running because they've been relying on name recognition and customers who can't be bothered to move.

Unfortunately, I think Pair is in the latter category.

Businesses that don't change with times do die a slow death. I have no idea how you've come to the conclusion though that Pair is in that category - would you like to clarify?

Then it's weird you're consistently trying to rebut complaints about Pair in this thread.

Did you even read what I wrote or do you just like to troll when someone says something that is contrary to your point of view?

The Akamai acquisition has had me kind of stressed to be honest. Here in Oceania there aren't a whole lot of established "cloud" hosts to pick from that host locally.

Linode is fantastic and I have loved them for 10+ years. Digital Ocean doesn't host here, nearest is Singapore. Vultr does but I have had poor experiences with them. The other options are AWS and Google which are potentially unbounded expenses which is hard to sell to clients that already see $12/m (aud) as a cost sink, no matter what the server+service will save them in seat time.

There are of course local hosts but they're either oversold cpanel hosts or comparatively expensive (3-10x the price for the same hardware).

I guess it just feeds from my existential dread of being a small fish where if any one of these upstream providers decide "oh, no we flagged that account because of [black box reason], sorry!", you can be out a lively hood with no real recourse.

Guess it just kinda sucks to be at what feels like the end of the long tail of infrastructure.

Just curious, what caused the poor experience with Vultr?

Not OP but just a guess, vultr was founded 2014 and linode is 2003. Obviously not a proxy for quality, but it’s one data point as to potentially why.

Was going to ask the same. I’ve used vultr for years and found them rock solid, and have had good support when needed.

They also just opened a DC in Melbourne.

These guys are in Australia amongst other locations: https://www.ovhcloud.com/

A caveat with them is that they're really trying to be price competitive, and can sometimes go overboard (like design their own custom datacenters and servers with watercooling, and skipping some things like fire suppression systems, and using cheap material such as plywood). Having a backup with a different provider, somewhere else, is always a good idea.

If you're very clever, watercooling is also the fire suppression system!

Possibly too clever.

Unrelated to the Linode thing, but in 15 years building and selling SaaS products, I've encountered a number of customers who nitpick prices or try to haggle. I'm not talking about non-profits with micro budgets, rather very profitable businesses.

Nowadays, whenever someone goes there, I just change my usual lighthearted tone to a serious one and say "Yeah, our prices are firm, if that's outside of your budget it's better we just move on". They 180 real fast. Every. Single. Time. Then they never mention prices again.

In the procedure list, someone included the line "Try to get a discount".

When it works, it works.

When it fails, it didn't work, but the next step is not "Quit" but "Continue".

And when it works, it contributes to the business being profitable.

(Kind of rhetorical, but interesting to me)

If you ever plan on colocating a 1U box somewhere, count me in for nibbling a bit off the end of it for $20/mo :) (I'm in Sydney, and seriously, seriously want my latency back)

I did a bunch of digging a while back and it looks like the "simply cannot go any lower" baseline is $250/mo.

Off topic: any updates on Linode's Bare Metal service?

It's been 2-years since it's been announced as "Coming Soon" and I haven't seen any updates.

https://web.archive.org/web/20201116142627/https://www.linod...

https://www.linode.com/products/bare-metal/ —still prefaced with "coming soon" but it looks like you can "create your build".

I gave up. I submitted an inquiry on their sales form for dedicated and got no response, I contacted their support and got "we'll let you know." They've at least added nvme storage for instances, but in at least their Dallas data center the iops suck. In Newark it's a bit better.

If you've got a bunch of servers at a linode data center and want bare metal ones, I ended up just looking at what other hosts are peered with their nearest exchange. Our servers are in Dallas and Vultr, Equinix via IBM Cloud, and Servermania all had peering to them through a local exchange at infomart or nearby. We ended up moving database servers to those bare metal services and connecting over wan to our linode ones. Latency less than 5ms.

Linode also contacted us to ask why our monthly usage halved. We told them, they offered no alternatives.

Same experience, no response. I tried contacting them both a year ago and about 6 months ago and never heard back.

I went with Vultr (as you mentioned as well) and couldn't be happier.

A bare metal data point, since I just went through the process of finding an alternative to Packet.net since they were acquired by Equinix : try Hivelocity. My experience with them has been so far so good.

Limestone networks (http://lstn.net) offers great bare metal services. I used to work there 7-8 years ago and they are a no frills, no bs company. They will give you a good deal also :)

Yep. I use their bare metal cloud and it connects to our whmcs so we’re able to resell it. I did get a better deal than they advertise by asking.

I think "Outage" as stated is a bit over-the-top. We run significant operations in most of these DCs, and we have seen zero impact. YMMV of course, but I think it's a big disingenuous to say this is a full outage.

It's a full outage for me -- wasting quite a lot of time. It's been 7 hours so far. Hopefully it won't drag on longer.

Who knows how many people it's affecting?

For us, too.

Mine in Dallas became inaccessible around 7:20am PT and came back around 7:50am.

For what it's worth I have around 25 Linodes active, mostly in Newark but a few in Atlanta and Dallas as well.

Of those I had two in Newark which actually had their leases expire before we were aware of what was going on and then found three more (two Atlanta, one Newark) once we knew what to look for which were failing to renew. All five of those machines already had network helper enabled as Linode suggests but were unable to get a lease even with manual operation of the DHCP client until they were rebooted.

Linode London has been experiencing "greater than normal" intermittent packet loss to Akamai network which they use to peer with other player such as Cloudflare in LDN for nearly two days - before they peered with CF via Linx if I remember correctly. It isn't just isolated to UDP customers. Currently it's fairly stable so for most it's hard to notice but certainly uncomfortable how far they want to drag this problem on - see status.linode.com for how grim the LDN situation is - their support said we'll tell you in a few days what it was once we fixed it. i.e. it's not the rock solid of the last 7 years and their support are largely deaf to how unacceptable this is based on previous standards.

to whomever is at the top of the automation repo commit log, i wish you a safe and speedy recovery. Enjoy this friday, that it may succor a distant memory of fridays past when the latest build didnt break production.

Amusingly, I am trying to log in to the dashboard to reboot my nodes, but it requires a verification code which was sent to my email, which is hosted on Linode.

Consider signing up to a service like MXGuardian that your MX records can point to, and that then forward email to your server. This will give you an “emergency inbox” that you can use in cases like this (might also help with spam depending on how you are currently dealing with that).

What happens when MXGuardian goes down?

My sites in Newark are up. The situation is not as alarming as the submission title would suggest. They've updated the status page to indicate that it is limited to an issue with DHCP.

I had 6 servers out of about 50 go down, all in Dallas/Atlanta even though most servers are in Newark. IPv6 connectivity seemed to remain up for all of them, though.

I can still login to the backend and to my vm through Lish but my websites are down. (I'm in Dallas)

edit: followed the instructions in this link mentioned upthread and rebooted, that solved it for me https://www.linode.com/docs/guides/network-helper/#individua...

We ended up rebooting our server, that brought everything back up again.

If it's DHCP and you know your IP (probably because it's your A record) try manually forcing the IP.

That’s too bad. Which other vps hosts are folks using these days besides aws/azure/gc?

If anyones thinking of switching to digitalocean, I’m currently switching linode from digitalocean who had an unforgivable, catastrophic loss of vm.

Digitalocean also prevebts customers from downloading disk image backups offsite for any kind of adult DR. Linode lets you download backup images right from the admin console.

I was thinking on moving to Hetzner (from Digital Ocean) - now that they have/will have a data center in Virginia....but i have heard iffy things about hetzner (some people are happy, others not so much). I should clarify, i'm only thinking on moving due to costs (not because i have encountered any ops or support issues). The stuff that i host is not as essential, and could do with slightly lower costs. ;-)

I'm using UpCloud who are good, but a bit more expensive than Linode or DigitalOcean.

Normally I'd agree, but... Have you seen today's status ? https://status.upcloud.com/ 6 hour downtime and no updates

For what it’s worth, I’ve been a Linode customer for about 12 years and have never had a problem. This current issue does not even affect me as my boxes are on static IPs. My setup consists of a number of Linode VPSes, a single Hetzner Box in Germany, and an S3 glacier bucket (the only Amazon service I use) for backups. It’s probably not optimal, especially for the business case, but it works for all my hobby and side projects.

For sure.

I’ve had linode for a very long time.

Not leaving them because of how good their tech support is or was to help me early on in my career, but they can’t be a point of failure either.

I’ll just get to see how easily a hybrid cloud might be orchestrated like yours. Things are drastically easier to learn to setup and maintain like you have said compared to 5 or 10 years ago.

Docker is docker, and traefik is traefik.

> an unforgivable, catastrophic loss of vm

deets?

Deleted everything despite there being a valid cc with plenty of room on the card.

They admitted they didn’t charge like they should have.

Hetzner looks good.

Thanks I’ll revisit them. Had a lot of dedicated servers in my past and turned the self hosting part of my brain off

I have also heard iWeb is or was a good choice for value. Montreal data centres can be a well kept secret.

Hetzner's dedicated servers are unbeatable in pricing. Especially the ryzen 5 3600x version.

Thank you I’ll have a look

I found my Linode-hosted site was down this morning. I logged into the Linode console, rebooted and it came back up.

Generally I've been very happy with Linode over several years.

Mine in Atlanta are up. Already had the 'Network Helper' enabled.

3 minutes of monitoring? That's enough to declare the issue over?

Downtime for me lasted almost 4 hours.

Seems like more and more outages are happening every day...tsk tsk

Shameless plug - we're helping orgs fix this and get better clarity:

https://www.awareops.com/

Shoot me a message mb @ awareops dot com or go ahead and sign up.

Reply on news.ycombinator.com