Linode Turns 14 | Hacker News Reader

Linode Turns 14

blog.linode.com

142 pointseatonphil67 comments

I used Linode for a really long time, hosting all sorts of personal projects over the years. A few Wordpress setups, some PhpBB forums, and other random small projects that I worked on. It was always nice having a box I could SSH into and do basic "is_it_down" ping/traceroute/nmap scans of things I was trying to reach.

Since all of my dynamic based sites are dead and I'm just on static based sites now, I just threw it all into Google Cloud Storage and host it from there (which greatly reduced my money hosting cost).

Overall Linode did a great job hosting for me, especially considering the price. But moving to "cloud" providers where I don't have to worry about keeping things upgraded and secure was worth the move to me. I enjoyed it for a long while, but there are easier to maintain services out there now.

I think you're right that Linode could be overkill if all you have are static based sites (although presumably you could host them all on a single $5/mo. instance.)

However, I find Linode's interface far less opaque and easier to use than Google and Amazon's for ongoing site management. I can also get a response from Linode in a few minutes when I submit a help request. Google's Cloud offering must have decent support, though I only know their reputation for one-on-one support across their other offerings which has traditionally been abysmal.

I loved Linode for their support. When I had issues, they were always good at responding to me and getting things resolved. AWS and GCP do have support, but you have to pay extra for it (sadly).

The other nice thing about Linode was the free upgrades. When they did fleet-wide updates to CPU/Ram/Disk, you could go into their UI, click a button and be migrated fairly quickly/easily.

I also agree that GCP and AWS are a lot more complicated, as they are vendor-specific ways of solving certain problems, which can be solved normally with Nginx and some configs (which is portable across any VPS/dedicated-host provider).

I'm not sure if they've recently changed something, but I just started using Google's Cloud Platform and the interface is really easy to use. In just a few clicks you can provision a VM with any amount of storage and then click a button to SSH right to it - key management is handled on the backend seamlessly. Compared to other "cloud" providers I found it the easiest and it allowed the most customization.

I've been pretty happy the handful of times I've had to interact with Google Domains' support. But for gmail, it was all but non existing.

Likewise. I ended up switching to Google Compute because Linode didn't have block storage when I needed it, and I don't plan to head back at this point. But they were the final destination host for my first startup after trying numerous less professional services, and I'll always remember Linode fondly.

I really don't understand how you could see Google or Amazon as upgrade comparing to Linode. Money for $5/mo shouldn't be the issue and it is infinitely simpler to just put things in folder then messing with Amazon or Google.

That is for static sites, for more complex sites where scalability is important, I think these offer advantages.

I've been with them since 2009. I know the current recommendation from security-focused people is not to use them, due to their mishandling of past events. However, I've found them generally to offer a higher quality of service and support overall.

Tried to check out their new Linode Manager, but: "The Beta Manager and API V4 are not available for legacy pre-pay accounts. Please convert to Hourly billing." Shucks. Guess I'll have to give up on the idea of an annual discount.

I hope the new API fixes the security issues they've had in the past. We'll see.

I too am bummed that the new stuff they're introducing doesn't work for pre-pay accounts. That annual discount was quite nice, and it's a shame they don't have an equivalent with their new billing.

Linode has a weak track-record with regards to security in my view.

I think this Glassdoor review sums it up pretty well: https://i.imgur.com/sJd56AT.png

Linode has been hacked at least 5 times¹ in the past decade, and EVERY TIME they try their hardest to downplay the incident and distract attention from it. Avoiding any public communications at all if possible.

This HN comment from a PagerDuty employee is also particularly damning https://news.ycombinator.com/item?id=10845985

¹https://news.ycombinator.com/item?id=10845278

While you and the sources you link to are generally correct, you should probably disclose your relationship with Linode when you make these comments -- namely that you were involved with, and successfully prosecuted for, one of the more serious of the five. You also lost any semblance of moral footing when you dumped the customer data that you extracted to the public. Reading your comment with that context changes it slightly; given that since said prosecution you've taken every opportunity to speak negatively about Linode that you can, I feel it's important context to have. I know you've spoken openly in the past about it, but a lot of folks reading now will not have seen those threads.

I'm not saying I disagree. To prove that, for the record, I informed Linode leadership that ColdFusion admin being world-accessible was a bad idea several years before you and your 'team' were even aware of Linode, and I and another employee proposed replacing ColdFusion long before you flippantly suggested that it doesn't take years[0]. (Try running a hosting provider at medium scale.) I've been just as critical long after leaving Linode as an employee -- I saw oversights like mgreb:mgreb1221 being a full-admin user on world-accessible db1.linode.com for many years with the password in crons, for example -- so please don't interpret me as disparaging your opinion, only your trying to have it both ways with a very clear conflict of interest, and turning your takedown of Linode into a potential annoyance for me and thousands of other people. And yes, I know the key passphrase that you never recovered, despite your public assertions to the contrary.

[0]: https://news.ycombinator.com/item?id=10845462

>While you and the sources you link to are generally correct, you should probably disclose your relationship with Linode when you make these comments -- namely that you were involved with, and successfully prosecuted for, one of the more serious of the five.

I was successfully prosecuted for hacking coldfusion things in general, that list included Linode and over 50000 others. There was a pending investigation relating to Linode specifically, but I'm pretty sure that's hit the statute of limitations and can only assume that there's no intent to prosecute.

>Reading your comment with that context changes it slightly

You're right, it does. I considered linking directly to https://news.ycombinator.com/item?id=10845278 but figured I could deliver the information in a more concise manner, without quoting myself.

I made a mistake in not immediately clarifying my relationship here.

>given that since said prosecution you've taken every opportunity to speak negatively about Linode that you can

Since long before that actually.

>And yes, I know the key passphrase that you never recovered, despite your public assertions to the contrary.

What are you referring to? The credit card decryption key? I assure you we most definitely got that, if you want I might be able to dig up old IRC logs from police documents with plaintext credit card information of several linode customers. Jennifer Emick, Robert "xnite" Whitney and Ryan Cleary as I recall.

> I was successfully prosecuted for hacking coldfusion things in general, that list included Linode and over 50000 others.

This is a significant exaggeration, but I'm not going to dox you. Suffice to say, I'm reading your case as we speak, so stop trying to mislead people. (Good luck with that visa.)

> What are you referring to? The credit card decryption key? I assure you we most definitely got that

Then name it. I have the one that protected the key you retrieved memorized on account of my duties (I had to enter it every time I restarted the application), and they've changed it since then, so it's a no-op. It's not brute forceable even by nation states, so you'd have had to extract it by means your entry vector did not give you. Hash it in your reply if you want.

I've always suspected you had former employee help, and I'm pretty sure I know not only exactly who, but also their motive for assisting you and "HTP". If you do have the passphrase to that key, I can almost promise it was from your assistance. I eagerly await your reply, since the passphrase is a sentence, and once you know it the gist of it is easily communicable without consulting logs.

>This is a significant exaggeration, but I'm not going to dox you.

It really isn't. The big number in that charge is referring to coldfusion shells the local CERT was supposedly able to "validate".

At best you could blame the prosecutor for exaggeration, which I entirely agree with, the court however accepted the prosecutors claims.

>Good luck with that visa.

Haven't actually had visa issues anywhere since then :) Good passport I guess.

>Then name it. I have it memorized, and they've changed it, so it's a no-op.

No clue, people within the group disagreed on carding so the person who got it didn't post it on the channel, only dropping the cards of "interesting" individuals. I guess I'll go looking for the logs though.

>It's not brute forceable even by nation states, so you'd have had to extract it by means your entry vector did not give you. Hash it in your reply if you want.

As far as I know someone wrote a quick coldfusion script to extract the decrypted key from memory.

>I've always suspected you had former employee help, and I'm pretty sure I know not only exactly who, but also their motive for assisting you and "HTP".

This was certainly not the case.

>If you do have the passphrase to that key, I can almost promise it was from your assistance. I eagerly await your reply, since the passphrase is a sentence, and once you know it the gist of it is easily communicable without consulting logs.

Don't know, and I'd assume getting coldfusion to spit out the decrypted key is much easier than the decryption passphrase that's only needed once.

And in any case your approach seems fundamentally flawed. Even in some parallel universe where we failed to extract the key, we'd still have been able to use your existing decryption code and the in-memory key to decrypt all the cards in the database.

> I guess I'll go looking for the logs though.

I haven't typed it since 2011 and still remember it; it's a stupidly simple English sentence that is almost impossible to not memorize. I also note that you and your group were challenged for it in 2013 and declined to provide it. But sure, go consult logs. I'll hold.

> This was certainly not the case.

I know the network and channel where you two hang out, so OK. Keep lying to HN, I guess. (Why would you?)

Seems disingenuous at best to quote that out of context.

>I know the network and channel where you two hang out, so OK.

Then spit it out, name the network and channel, don't make vague accusations. I'm open about my beef with Linode, but what's your beef with me?

If this is supposed to be more than a pathetic attempt at discrediting me with false accusations, please follow through.

You first, "Ryan." My beef with you is that you (a) hacked Linode, (b) dumped everything you found online, then (c) spend your days in every Linode thread trying to shit on them while pulling wool over HN's eyes as some kind of moral high ground just "obligatory warning" a bunch of people about your own fucking actions. You are not the moral high ground. Please stop pretending to be as some kind of authoritative voice. I've watched you do this for years, and it's starting to grate on me. I've disclosed I'm a former employee every time I speak about them, and I've also pointed out that I harbor no ill will for them: the people who run Linode, despite making mistakes, do not deserve this stalkerish behavior from you.

Move on with your life. You hacked them, you got busted, and sitting around like this four years later is just indefensible.

Edit: I'm remaining vague so as not to dox you, given HN guidelines and my own personal ethics. I don't have any stake in discrediting you (you do a fine job of that yourself), but you're welcome to spin it that way. Also, your clock ran out on being uniquely positioned to comment in 2014. (As did mine, though sooner.)

>(a) hacked Linode

Fair.

>(b) dumped everything you found online

Not even true. Neither Linode customer db or source code were publicly released.

>(c) spend your days in every Linode thread trying to shit on them while pulling wool over HN's eyes as some kind of moral high ground just "obligatory warning" a bunch of people about your own fucking actions.

Mostly not my actions, but I seem to be uniquely well positioned to give such advice given my that I've seen the insides of Linode.

>You are not the moral high ground. Please stop pretending to be as some kind of authoritative voice.

I'm certainly not the moral high ground, but I've certainly seen for my own eyes just how much of a mess Linode is.

>Move on with your life. You hacked them, you got busted, and sitting around like this four years later is just indefensible.

I don't really care about any of that. To be fair I'm more annoyed with a bunch of my time being wasted because some guy with a thick Australian accent (i.e. obviously not me) swatted them.

But this isn't my personal blog, hit me up on IRC and we can chat.

I remember first using your service almost a decade ago and then, drifted over to DO over the last few years.

Good to know you folks are going strong - all the best.

One thing I would really like as a 7 year old Linode customer is something like Amazons VPC, because that could really ease a lot of my deployments.

They do offer private lan networking, but that is shared with all the other customers, if I don't trust the internet, I won't trust other Linode customers.

Does anyone have experience with a hosting provider that offer something similar?

Digital Ocean just launched that: https://blog.digitalocean.com/cloud-firewalls-secure-droplet...

It doesn't quite imply a trusted network. It controls the set of hosts that can communicate, but doesn't say anything about the possibility of interception or tampering like a VPN or VPC does.

I actually looked at that, and thought it didn't solve my problem, but I skipped this part: "and even sets of Droplets by using Tags.".

(IBM) SoftLayer runs a separate public and private network; and private is segregated by customer. Within the datacenter it's a complete separate system, connectivity between datacenters seems shared by public and private (private generally gets priority when there's a capacity crunch, though); public bandwidth use is metered, and private is unmetered. I work for one of their customers.

"We’re building our own transit backbone across the planet. More on that in an upcoming blog post next week."

This is interesting... Leasing point to point connectivity like this is surely very expensive for a VPS provider, I understand non-connectivity players like OVH and Google Cloud doing this but Linode?

What do you mean exactly? Linode is in the same facility in, for example, Frankfurt as everyone else (DigitalOcean, Vultr, etc.) and DO and Vultr both have the better network, so I don't quite understand why Linode is special in your eyes?

Being in the same physical location doesn't mean they have the same connectivity though. With colo cages, once you get big enough you get to choose your upstream providers.

I get 15 ms better ping to Frankfurt Linode compared to Vultr, for example.

They all have their own network. They even have the same transit providers in Frankfurt (Linode and Vultr both use Telia for example). All three get to choose their upstream providers, your argument is invalid. In my tests DO had the best network even though all three save a lot of money by not getting transit from DTAG or UPC, which is a bummer for german customers.

Just compare: http://bgp.he.net/AS63949#_peers to http://bgp.he.net/AS200130#_peers

And you will see that DO alone in Europe (compared to Linodes global ASN) has many more transit providers and peerings.

I mean I didnt think Linode had the financial means to build their own global backbone, I dont consider Linode special. Vultr relies exclusively on transit providers, DigitalOcean uses a mix of transit providers and often direct connection to popular local peering fabric or two. Building your own global backbone like HE, OVH, or any of large transit providers like NTT, Level3, Cogent etc must be a massive cost.

Worth noting that Linode didn't even have an ASN until very recently.

What are the issues with Linode not having an ASN?

My comment was more to indicate the overall state of their own network, which didn't exist at all until very recently.

Linode may be 14 years old, but they've only been running their own network for a year.

If they don't have an ASN. It also means they don't have their own IP address space and could not build a transit network.

That's not true. Linode had numerous allocations, including up to a /16 (if I recall), before receiving an ASN. They simply announced them from datacenter transit, which has pros and cons.

> Block Storage Volumes are highly available, fast, and inexpensive – $0.10 per GB (free during the beta).

This is significantly more expensive than AWS S3 or Google Cloud Storage.

The last time S3 charged similar prices was in January 2014: https://aws.amazon.com/blogs/aws/aws-update-new-m3-features-...

edit: I'm keeping this comment up in case other people have the same confusion I did.

I think this is a bit misleading. In my experience, EBS/S3 and GCS expenses are dominated by data transfer costs. Linode is providing 1 to 20 TB data transfer free with the compute. While comparing, we'd have to compare (one Linode plus block storage costs) to (one EC2/GCE node plus EBS/S3/GCS storage costs plus network transfer costs).

This is more like EBS on AWS, which is also $0.10 / GB-month for gp2 volumes.

That's pretty standard pricing (look at DO block storage or Vultr's offerings). Keep in mind that this is true block storage, not document or object storage. You can mount object storage, sure-but it's always going to be less reliable than actual block storage.

Block storage is completely different from object storage (S3). You're comparing apples and Volkswagens. This competes with ebs, except with more redundancy.

I remember being a poor student in the 00s, looking for hosting providers and Linode was a new cool kid on the block (prohibitively expensive for me though). Now a lot of things changed, and I can easily afford a VPS now, but Linode is still around and still is one of the better options. DigitalOcean is basically twice as expensive.

10+ years customers here. Except that my VM crashed once, all the years have been reliable.

Their new control panel brings them up to par with Digital Ocean's in my opinion: https://cloud.linode.com/linodes (requires login)

I use a Linode instance for a personal OpenVPN server and the bandwith for me is at lest 2-3x bigger. This is a huge deal for me for the same price.

I wish Linode had ARM64. I am so done with Intel bullying other corps, I moved my work to Scaleway for that reason.

Linode has an INCREDIBLE support staff. Like, superhero action squad to the rescue as soon as something goes wrong incredible, they're just that good.

Please get some real boxes though! I really want to come back!

This is the reason I stay with Linode. The price/service ain't that much different than elsewhere but having staff who are experienced AND responsive is a killer feature.

I've been with them since 2009 and can confirm that every time I reached their support I've got nothing but good answers and quick action.

Worth the downvotes:

Linode Turns 87178291200

Hmmm... I think your calculation is off...

    var now = new Date();
    var then = new Date();
    then.setYear(then.getYear() - 14);
    console.log(Math.floor((now - then) / 1000))

    // 60400079994 (seconds)

It's a pun on "14!".

filed under "not-so-shameless-plug".

14! is quite old. 87178291200 years in fact.

You missed an opportunity for a good pun.

Linode is fully-in the bratty teen years.

What is the intended sentiment behind this comment?

That the company, were it to be a human, would be a 14 year old teenager. 14 year olds, as you know, can be brats due to various reasons (hormones being first among them).

It's just a lighthearted comment about the age of the site/company.

I get that it wasn't funny...glad someone at least saw the context.

Obligatory Linode warning. Linode has a history of putting their customers in very uncomfortable situations.

Here's a few HN threads on the previous disasters:

https://news.ycombinator.com/item?id=3654110 Compromised Linode, thousands of BitCoins stolen (2012)

https://news.ycombinator.com/item?id=3655137 Linode Manager Security Incident (2012)

https://news.ycombinator.com/item?id=5552756 Linode hacked, CCs and passwords leaked (2013)

https://news.ycombinator.com/item?id=7086921 An old system and a SWAT team (2014)

https://news.ycombinator.com/item?id=10825425 Linode DDoS continues – Atlanta down for 16+ hours (2016)

https://news.ycombinator.com/item?id=10998661 The Twelve Days of Crisis – A Retrospective on Linode’s Holiday DDoS Attacks (2016)

https://news.ycombinator.com/item?id=10845170 Security Notification and Linode Manager Password Reset (2016)

https://news.ycombinator.com/item?id=10806686 Linode is suffering on-going DDoS attacks (2016)

Edit: Guess someone linked this on the Linode staff channel :) Hi guys!

Around November in 2015 the IRC network I host was hacked due to a breach Linode had already been informed of (https://news.ycombinator.com/item?id=10845985) - I had a completely unique username, password, and I even had 2FA enabled. Linode support refused to work with me, advising me to "secure my 2fa device better next time". Longevity of a company like Linode clearly isn't indicative of good business practices, only that they are the cheapest option.

Why was this downvoted?

Probably because of https://news.ycombinator.com/item?id=14572970 ?

> "While you and the sources you link to are generally correct, you should probably disclose your relationship with Linode when you make these comments -- namely that you were involved with, and successfully prosecuted for, one of the more serious of the five."

The comment you linked is newer than the one you're responding to, probably not.

Why do you hate then so much?

They are just another hosting company, I'm sure your can find holes and dirt on all of them?

Why not attack an organisation that is "more evil" ?

First of all I need to clarify that I'd likely be making these posts even without my personal interactions with Linode, just as I'd advice against using SSL 2.0 or gasp ColdFusion if I saw someone using them.

I'm certainly not alone in recommending against Linode, any sane security person will share the sentiment.

>Why do you hate then so much?

I don't really "hate" Linode, but I certainly don't wish them luck.

Why?

They wasted a bunch of my time because some guy with a very thick Aussie accent prank called them and the police while introducing himself as me.

>Why not attack an organisation that is "more evil" ?

I can do both, although I'm hardly "attacking" Linode. I'm simply pointing out their very poor track record, which is an indisputable fact.

Why advice folks away from a hosting provider that covers up known breaches for months? https://news.ycombinator.com/item?id=10845985

Or one that tells their employees to lie about breaches? https://imgur.com/sJd56AT

Or one that refused for years to spend a couple of minutes thinking about security, until it started to actually hurt their bottom line? As jsmthrowaway points out, Linode was aware of serious security issues for years and did exactly nothing to protect their customers. https://news.ycombinator.com/item?id=14572970

If not for the info you provided I may have wasted my time with them.

Honestly, I was wondering if Linode staff or people on "their side" downvoted the post since it didn't seem downvote worthy. It's only fair that their track record follow them, especially their free press. It wasn't a rant or an accusation or anything personal. It was just a list of their past actions and what others had to say about it.

If you want trustworthy cloud hosting look at EC2 and GCE. Avoid digital ocean, they have a long history of account hijackings due to "social engineering" attacks.

Otherwise look at dedicated hosting, preferably colocation but just renting servers can be cheaper and isn't so bad. Encrypt your drives, either disable your IPMI thing completely or ensure that it's only accessible over a VPN by you only and not your host. OVH (and their other brands) might be a decent cheap choice for dedicated hosting that should be very competitive in Linodes price bracket.

Reply on news.ycombinator.com