Linode Security Advisory
blog.linode.com
blog.linode.com
I find this update very hard to follow. Can someone tell me if I'm misreading it? I'm going to quote it twice, and then attempt to summarize:
After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation.
and then:
The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.
Linode’s security team did discover a vulnerability in Lish’s SSH gateway that potentially could have been used to obtain information discovered on December 17, although we have no evidence to support this supposition. We immediately fixed the vulnerability.
Here is my read of what this says; I'd like to know if I'm wrong.
"One of our customers got owned up in July, and gave us an attacker source address within Linode. We pickled up the attacker's host. In December, we examined the pickled host, and found secrets related to the way we store 2FA credentials, indicating that our credentials database may have been compromised. In conclusion: we have no idea how that could have happened."
Am I missing something else?
The findings of our security partner’s investigation concluded there was no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials. Furthermore, the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.
Did they get compromised or not?
1. Their security partner (whoever that is) didn't see the Lish vulnerability, either because Linode's security team had already fixed it or because they just missed it.
2. Their records weren't sufficient to show the breach itself.
no evidence of abuse or misuse of Linode’s infrastructure that would have resulted in the disclosure of customer credentials.
I feel like I must be misreading something. Didn't they say earlier that they found secrets from their account credentials database on a customer instance that was used to attack (apparently) PagerDuty? That's not "no evidence of abuse or misuse of Linode’s infrastructure"?
If you just assume they are attempting misdirection it makes perfect sense for them to insist things are fine.
I'm speculating, but having been a fly on the wall at this kind of meeting before, here's my theory of how this went down:
Manager: "So somebody got the key to generate one-time password tokens for PagerDuty. How did that happen?"
Engineer: "I have no idea."
Manager: "What about that Lish vulnerability? Could it have been that?"
Engineer: "Could have been."
Manager: "We don't know?"
Engineer: "I mean, they could have gotten it that way, but we don't have logging for security vulnerabilities, because, you know, we would just fix them instead of logging that they occurred..."
Manager: "Okay, I want a second opinion. Time to bring in $SECURITY_PARTNER."
$SECURITY_PARTNER: "What Engineer said. And here's a bill for our time."
Manager: "What about if the customer screwed up and lost a device? Could it be that?"
Engineer: "It could be that."
Support: "PagerDuty said something about wiping their phone."
Engineer: "If they did lose a device, it would be indistinguishable from the evidence we have."
Manager: "So, maybe we weren't hacked after all?"
Engineer: "Maybe not."
Manager: "Any other theories?"
Engineer: "I just want to go on record... AGAIN... that I don't trust our ColdFusion infrastructure at all. The people responsible have all left, and none of us understand it. We need to rewrite in a cooler language, like Python. And hire some actual security people."
Manager: "Sigh. We've gotten enough flak over this that I guess I'll put it in the budget. Okay, good meeting everyone. I'll write this up in a blogpost, because we need to show transparency. That way customers are at least as confused as we are."
That would indicate that Linode's password hashes may have been cracked offline.
> After examining the image from our July investigation, we discovered software capable of generating TOTP codes if provided a TOTP key. We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code. Though the credentials found were unrelated to any of the unauthorized Linode Manager logins made in December, the discovery of this information significantly changed the seriousness of our investigation.
so, it was an inside job then?
When it comes to the security of who's hosting my servers, I want a little more reassurance than they shouldn't have access. I need to know that they don't.
The blog post states they're "working towards" these changes, they're not currently in place. It's fairly unlikely that they're using the same secret key as the one they found on the server, but it's fair to assume that they are still using salted SHA-2 for your passwords and the same 2FA setup right now.
They likely won't roll out the major changes until they roll out the "new and improved" Linode dashboard they're coming up with.
Seriously, in one paragraph they state their TOTP keys were taken, in another they state nothing is wrong because they can't figure out how an attacker might have taken those. WTF?
On the other hand they've had security issues fairly regularly and their response to the DDoS was pretty poor.
That said if I was a cynic I'd say they probably have one of the best setups for dealing with future attacks (old joke about never firing an employee who made an expensive mistake because he'll never make that mistake again) and finally seem to be taking security seriously, for me the list of changes all sound good (particulary open sourcing Linode Manager and tokenising CC's had to reset a card because of them once before).
We are slowly moving a lot of stuff back to a DC up the road but we still have some stuff with them.
I think for me the largest red flag was DO not even having bandwidth monitoring in place. The only reason there were no bandwidth limitations was because they were not even physically tracking bandwidth usage. How do you launch a hosting platform without something as basic as being able to monitor bandwidth?
I have been saying for years people should not be trusting Linode for anything important. The transparency and the honesty simply isn't there.
I've relocated my VPS to AWS following these recent discussions around security, and the same test suite now runs between 5 and 20 minutes, presumably based on what my neighbours are doing at the time.
It's a frustrating tradeoff.
Yeah, LiSH was exploited like 3 years ago...
Unless it's been completely redesigned it's probably still got heaps of vulnerabilities, screen wasn't designed for untrusted input.
>CC Tokenization: Although our investigation yielded no evidence of credit card information being accessed, we are taking advantage of our payment processor’s tokenization feature to remove the risk associated with storing credit card information.
Nobody thought about doing this when every customers card plaintext card info was taken years ago?
Anyway, the entire blog post is bullshit. It completely fails to address their previous security track record. A single hack isn't that big of a deal, but this has happened to Linode countless of times.
But it is quite surprising that someone was able to acquire the key for the token generation and they seem to have no explaination for it. And wow, they only started tokenizing credit cards now? And SHA-2 for password hashes?
THB, after reading this post my confidence in them hasn't really increased. It feel like: "We fucked up, but are not exactly sure why but will fix some issues nevertheless".
How so? Either they're 100% clueless or they aren't being transparent.
It feels like they have no clue at all how it happened, but want to fix some issues nevertheless.
I'm not sure if they thought this would bring confidence back. Because if so, they failed hard.
Linode has a long, long history of being less than honest and less than transparent.
They're moving on from them, but they're going to leave SHA-2s sitting there and wait until everyone logs in to upgrade to bcrypt hashes at rest.
Not getting a super competent vibe off of these folks.
In practice, I suspect that either the bindings for Argon2/scrypt don't exist or aren't easily adoptable given their use of ColdFusion. They do exist in Python.
Either way, it seems like a sub-optimal decision.
(Also last I looked Python has no good scrypt bindings.)
https://paragonie.com/blog/2016/02/how-safely-store-password... https://news.ycombinator.com/item?id=11118720
they should be adopting Argon2
It currently isn't ready in large production. Efforts to stabilise the API are being spearheaded by someone apparently outside the project[1]. If you're reading this @lucab, thank you.
In the meantime, my Ruby bindings have been broken on three separate occasions due to API changes. You could easily say "Don't track master", but the one release has a tag of 20151206, and it's just an arbitrary a tag as any particular commit id. There is no branch from which you could apply "bugfix only" updates.
Two separate commits broke compilation. This commit[2] was a shambles.
Most importantly, they have commits going in two days ago that change the test vectors[3]. That means if you update your library, verifying existing passwords breaks. The hash identifier doesn't change ( in the way that bcrypt had $2, then changed it to $2a then $2y when they changed the algorithm) which means you can't just write an "upgrade hash" function. I can't find any documentation relating to this change.
It's important to note that none of this means your passwords are easily broken, or that it's insecure, which is the implication I often see thrown around when discussing Argon2 being "new".
[0] https://github.com/P-H-C/phc-winner-argon2/commits/master?au... [1] https://github.com/P-H-C/phc-winner-argon2/issues/61 [2] https://github.com/P-H-C/phc-winner-argon2/issues/87 [3] https://github.com/P-H-C/phc-winner-argon2/commit/37e031213e...
The way upgrade should work is that the user provides their password, which is verified with SHA-2 and then hashed with bcrypt and stored again.
In order to do this without people logging in Linode would have to bcrypt hash the SHA-2 hashed passwords and then keep doing that for all password validations.
It's not even remotely competent. This blog makes it clear they're not even sure how their secret key was stolen. These hashes could be walking out their backdoor as I type this. Keeping vulnerable hashes at rest is insane.
It would be far more competent to bcrypt the SHA-2s, so that at least when the hashes wander out the backdoor they haven't really found, peoples passwords aren't trivially attackable.
> In order to do this without people logging in Linode would have to bcrypt hash the SHA-2 hashed passwords and then keep doing that for all password validations.
No, they'd just have to replace Bcrypt(SHA-2(password)) with Bcrypt(password) once the customer finally logs in.
It's an immediate net upgrade to the resistance of at-rest Hashes to brute force attacks with zero downside.
My point was that if you have a system which can go straight from SHA2(password) to bcrypt(password) then the system must be storing the plaintext of the password, which would be very bad.
Yes, I understand that. It's just completely irrelevant to the question of whether or not it's competent practice to store vulnerable hashes indefinitely, awaiting customer log in.
Again, it is not a competent practice. Wrap vulnerable hashes in strong ones immediately; they're a huge liability to leave sitting in your storage even when you don't have evidence that there's a backdoor in your systems that you cannot seem to find.
> At this point in time all Linode should know is
> SHA-2(password) and they can't use that to derive
> bcrypt(password).
> ...
> In order to do this without people logging in Linode would have
> to bcrypt hash the SHA-2 hashed passwords and then keep doing
> that for all password validations.
Er no, forcing email password resets and blocking the old passwords is the only way you should ever handle breaches like this.
to the downvoter: do you think it's cool to fuck over every single one of your customers that hasn't logged in recently?
> we are taking advantage of our payment processor’s tokenization feature to remove the risk associated with storing credit card information
> we are hiring a senior-level security expert
I'm a PagerDuty employee and am the same individual who made this post on the last HN thread:
* https://news.ycombinator.com/item?id=10845985
Unfortunately, there are some facts in Linode's post that are not correct.
>On July 9 a customer notified us of unauthorized access into their Linode account. The customer learned that an intruder had obtained access to their account after receiving an email notification confirming a root password reset for one of their Linodes. Our initial investigation showed the unauthorized login was successful on the first attempt and resembled normal activity.
This is almost correct. Someone got in to our account on the first try. They knew the password and a valid TOTP token. Although, Linode's email isn't what notified is, it was our intrusion detection system.
>On July 12, in anticipation of law enforcement’s involvement, the customer followed up with a preservation request for a Linode corresponding to an IP address believed to be involved in the unauthorized access. We honored the request and asked the customer to provide us with any additional evidence (e.g., log files) that would support the Linode being the source of malicious activity. Neither the customer nor law enforcement followed up and, because we do not examine customer data without probable cause, we did not analyze the preserved image.
This is partially correct. We informed Linode that we saw suspicious activity within their network, and reached out to them to inform them. We provided any and all logs we had. We also informed them that we passed the info on to law enforcement, in case they wanted to proactively preserve the data. The knew we had no further information, and as such didn't ask for anything additional.
>On the same day, the customer reported that the user whose account was accessed had lost a mobile device several weeks earlier containing the 2FA credentials required to access the account, and explained that the owner attempted to remotely wipe the device some time later. In addition, this user employed a weak password. In light of this information, and with no evidence to support that the credentials were obtained from Linode, we did not investigate further.
The story behind the mobile device is totally incorrect. The user did not lose their device, the device had been restored (intentionally wiped) 9 months prior to the compromise. The user got a new device, and never set up MFA on their new phone after wiping the old one. The device was, and still is, in the user's possession. The device has not been powered on in a long while.
The user who was compromised was no longer in possession of their MFA secret. They deleted it, intentionally, with no backups existing.
If anyone here is going to be at Velocity 2016 in Santa Clara, or at Monitorama PDX 2016, I'll be giving talks on how PagerDuty was compromised back in July. This includes full details of how this happened, including the details of the mobile device referenced above. There are some details in my talk that don't line up with the blog post provided by Linode. :)
How many times has this kind of thing happened to them now?
I can't even begin to imagine how much money your customers could lose if PagerDuty infrastructure went down.
Keep in mind Linode had a pretty good rep at the time. I wouldn't second guess them on the call, as I probably would have made it at the time, too. They didn't dig in or lock themselves in with debt, and bailed when the time was right, too.
Are you able to elaborate on this? I understand you may not want to name specific vendors/products in the name of operational security but it sounds like in this scenario whatever is in place actually did its job.
* https://www.pagerduty.com/blog/security-monitoring-alerting-...
Most of this is still valid. There may be some differences as we've improved our configuration over time.
We use OSSEC for host-level intrusion detection. This fired off quite a few alerts as the malicious party began to log in as root on the serial console, amongst other things.
We also have supplemented it with other tools, such as an in-house wrapper around nmap, to alert us to hosts that don't match their expected network configuration. So when ports get opened incorrectly, someone is alerted usually within a minute.
I have been doing some research in my (limited) spare time to try to find a new provider, but I still have not made the switch from Linode.
Bah. More virt. Throw down some cash on a cage and fire up AWS Direct Connect, man. It's bliss, you get more choices out here than I did with us-east-1, plus you guys can afford it now.
We plopped in physical gear for a couple parts of our AWS infra a few employers ago and cut our Amazon opex by like, two thirds. Not helpful for your using Azure as a reliability strategy, but worth thinking about for your write-heavy databases, for example.
We operate all of our datacenters in a multi-master configuration across the WAN, so latency is something we need to be mindful of. We were also in the middle of an emergency situation that required a migration, so we needed a solution that would allow us to evacuate quickly.
In the end we decided that we wanted a provider who supported a VPC-like network configuration, and was roughly within the same latency profile as Linode in respect to US-WEST-1 and US-WEST-2.
If timing hadn't been a concern, we may have chosen differently. We felt we didn't have the luxury of time.
/edit never mind they are not as per another reply in this thread
Xeon E5504 - 45nm process, launched 2009
Core i5-2300 - 32nm process, launched 2011, discontinued 2012
I cringe when I see companies say this. As if we're supposed to feel like the "hack" was somehow more sophisticated than spearfishing or social engineering because there's feds on the case.
There's a hole in your security. Diligently look for that hole. If it's a mistake own up to it fully and apologize. Make your system robust. If you don't have the talent in your organization to do this, then hire more talented engineers. Compete with other companies for good people.
Regardless of the severity of the means, the fact is that this sort of attack is entirely illegal and involving law enforcement is a clear requirement.
For example, I run about 10 different sites off one Linode, but only one of them gets any substantial traffic. Still, in order to run that site, which works just fine on a $20/mo 2GB/2core unit on Linode, I'll push out about 150GB in outbound bandwidth a month, and require around 15-20GB in storage. Pricing that out on AWS, ignoring the free tier, I'm looking at at least $40/mo, so double the price.
These are hobby sites or side projects, so they are not necessarily mission critical, but it would be sad if they went down.
There's a serious benefit to getting a VPS when you're on the low end like that, where power and bandwidth per dollar are greatly maximized.
The main value provided by cloud hosting is easy scalability, not cheap prices.
You can get a cheap "High Availability" setup for $10 with servers on two different continents.
Oh, and the bandwidth is free. (Although, 100mbit.) You can max that line 24/7 for a month and pay nothing more than $5, on linode that costs twice as much that'd cost $600 just for the bandwidth.
If you need something beefier than Kimsufi, OVH also has their mid-range brand soyoustart (https://www.soyoustart.com/us/essential-servers/) with prices starting at $42 that'll just completely destroy any similarly priced cloud offerings.
I'm only focusing on OVH here because they're IMO as the biggest provider in this market they're also the safest choice. Leaseweb and Voxility might be worth checking out too.
Quick disclaimer: I'm not currently an OVH customer (their DC locations don't fit my use case) nor am I in any way affiliated with them.
- It's consistently faster on most metrics than other providers I've tested.
- It's full KVM and allows you to run any OS you like. As long as it's KVM virtualisable and you can install it from an ISO it's an option.
- It's present in a ton of DCs.
This is fine, but no remote jobs. I actually would have taken the job if they offered remote employment.
I've also only ready bad things about the executive level management. I asked the interviewer about it, they did seem to confirm my suspicions in this regard, but he did say that they had some great new technical leadership that will be driving the company forward.
Seems they really have very flippant executives, or it's just their CEO.
There's a lot more to the story, for sure. It's the worst of the bro culture, or at least it was when I left. At my next employer they looked over Linode employees as recruiting opportunities after hiring me, and came to ask me about potential candidates, and the only one they were interested in was genital rubber. I laughed and said I'd quit.
Also, I upvoted the comment because I think it should stay visible. I hope other users do that too instead of just downvoting.
There was also stabbing employees (to the point of requiring an ambulance) while fooling around with a knife, setting the building on fire more than once, and tormenting other employees who he didn't like. All of that was tolerated and dismissed by management, specifically Chris Aker and Tom Asaro, which should tell you what you need to know about ever working there. One of Linode's early forays into hiring women ended very, very badly, and the only one who made it from that time period was forever marked by the experience.
And since he outed himself (I was careful not to), it's worth pointing out that the same individual now sits on the school board where he lives. Food for thought.
I required several years of counseling to recover from working at Linode, and I am no longer under any legal obligation to keep that a secret. I'm consulting with an attorney regarding the best way to tell my story without incurring legal liability, because Linode nearly killed several people who are very dear to me.
I've certainly worked in similar atmospheres. I'm glad you've put it behind you.
Also, all the people who quit Linode and ran to this coast after I left have kept me very apprised of what working at Linode is like TODAY. I still communicate with your colleagues quite regularly, too.
It is important to reiterate here, as I have before, that I wish Linode no ill will. I'm becoming more comfortable with calling spades spades, but I do not wish Linode to fail. I actually hope a lot of this stuff can be fixed, whatever that entails, but I have a serious gripe with some events that have transpired since 2011, from security to personal. If Linode would start being a little more honest about their gaping security troubles, and not rely upon people like me and Tim who actually know the truth to just shut up about it (I'm getting braver as Tim does, and I appreciate how willing he is to not let PagerDuty be tossed around by Linode's deceit), I'd be a bit happier. We're crossing into knowingly compromising the safety of the Internet, PII, and a number of production infrastructures that still run on Linode in some cases, and I do care about that.
And again, that tone of deceit is set from the top.
> And you know it.
That's unduly personal. Please remain civil.
I bit my tongue on you detaching this subthread because I've learned that moderation is opaque and largely not welcome to outside opinion, but I agree with Ryan up there and suspect you detached the thread to hide where I went with it. I'm fine with that (honest). Just wish you'd say that.
I've edited, regardless. Is that better?
I don't have the least opinion about Linode or "where [you] went with it". My concern is with civility on Hacker News. That's not an arbitrary line, though I'd never claim we make every call correctly.
The GP seemed to me merely to be saying that the company had changed since you left. That doesn't seem personal, nor a threat, but perhaps there are subtleties I'm missing.
We detached this subthread from https://news.ycombinator.com/item?id=11136707 and marked it off-topic.
tptacek asked about inconsistencies in the Linode advisory, I posted a screenshot of an apparent Linode employee claiming that they lie to their customers regarding security issues. (Which is a claim I am willing to personally back up)