Never Give Your Information To 10 Minute Old Startups
blog.ryankearney.com
blog.ryankearney.com
----
"if anyone's concerned about your AWS key, just destroy your IAM user and create a new one. that's what it was designed for."
----
In response to advice saying they should notify users by email:
"good idea. actually, we'll just wipe them and force new ones."
----
In response to RKearney warning people about just what exactly is exposed:
"in case you have issues with your AWS keys. RKearny's email: ryan@ryankearney.com https://secure.gravatar.com/avatar/f7d7b021fb488fe6a67ddb286....
If you make a mistake, own up to it. Honesty is the best key to building a business, and I'm sure they've at least lost the HN trust for any product in the future.
They should have thanked him, notified their users, done a thorough review of their own security, and warned new signups to only use IAM keys. Instead they got defensive, made excuses, and attacked the messenger.
Sorry, but no. There is absolutely no good reason while creating an user controller - even when writing the very first lines - to not check against current user on sensible actions.
Actually, I may argue that's the very first thing you do once the controller skeleton is set up.
def edit
@aws_credentials = current_user.aws_credentials
end
def update
current_user.aws_credentials.update_attributes(
params[:aws_credentials].slice(:a, :b, :c)
)
endBut that "contact Ryan if anything goes wrong" is grade A asshole.
This incentivizes people to fix things quickly and preserves the reputational value of breaking into things without researcher-vendor relations getting adversarial when you announce something like "I harvested a couple dozen of your customers' API keys" or "Here's an exploitation roadmap you can follow in your browser" in a public forum.
A policy of so-called responsible disclosure is a reasonable approach to take when dealing with an established product/service that contains a minor vulnerability, something potentially dangerous but unlikely to be exploited in the immediate future with serious negative effects.
In this case, we appear to have a new project run by people who don't know what they're doing, with a glaring vulnerability that had presumably already compromised 80+ people's sensitive credentials and in turn who knows what other sensitive information. Bringing it down as fast as humanly possible and loudly so no-one else gets damaged in the meantime is entirely justified in a case like this.
In his own responses, he says that he won't email the users! I can't imagine how upset I would be if this had my information. Access control for something like this is dead simple
If they're based almost anywhere in the US or EU, they might well have a legal obligation to notify at this point.
(I am not your lawyer, etc.)
In this case I think the public disclosure would have two effects: 1. Put current users of the product at risk. 2. Prevent people from signing up for the product.
So the question really becomes, does 1 outweigh 2, or vice-versa? (and the answer to that also depends on how cooperative & quick the company will be with a fix)
With responsible disclosure, only the original poster and anyone else who happened to figure this out would know. Now everyone does, and any random attacker can just go to the site and harvest AWS credentials from anyone signed up.
In a position where no choice of action/inaction is guaranteed to be harmless, I think limiting the damage is probably the most practical choice, and certainly a reasonable one. It limits the number of potential victims, and it also serves as a warning to those developing future sites that this sort of screw-up is not acceptable.
1) The founder's behavior in the other thread, including refusing to notify affected parties.
2) Such a simple mistake worries me about what else might be vulnerable in the application which is built to handle users' backup data, and for that reason alone, I think this article is extremely important right now.
Since whomever discovered the bug was able to access others' sensitive information, they have to disclose.
Don't trust the client.
The user ID in the URL like this is a giant "try editing me and see what happens" sign, even if you came with no intention of providing unsolicited pen testing. I seriously doubt just this one person noticed.
What's really surprising is that apparently the dev on this never viewed any porn on the internet.
If a backend is coded this poorly, it betrays irreparable and highly dangerous levels of idiocy, laziness, and lack of foresight in the ones who coded it. Everyone deserves to be informed of this blunder so they know to avoid this group like the plague.
Public ridicule and preemptive destruction of the brand is the only conscionable reaction.
They probably should have shut it down or disabled registrations once it got out until it was tested.
Sorry, but if you put a public site on the Internet, somewhere it can be discovered, and you are prompting people to put in sensitive credentials on that site, then you have launched for practical purposes. You should be implementing security measures accordingly.
If you're not ready for that and just want to show friends, it's not exactly rocket science to add basic HTTP Auth to the site, lock it to specific IP addresses, or any number of other trivial measures that would have prevented this problem.
Before ever putting a service up on the public Internet (service defined here as "accepts arbitrary requests" and "delivers arbitrary responses"), I would hope every human being that knows his way around a text editor treats user data like the Dead Sea Scrolls. If you store a row in a database, you then think of every way that an unauthorized party can gain access to that row and close each in multiple ways. I can recite dozens of cases where user data hasn't been treated with the respect it deserves (i.e., every single Bitcoin disclosure due to newer developers running sites that are handling money).
If people took user data more seriously than they do in general, we'd have a lot less leaks. Imagine if this had gone undiscovered and the service took off? Imagine how many undiscovered vulnerabilities there are in there, with this track record to start?
I can't sympathize with this at all. I just can't.
It should not have been on the public internet without access control for editing/viewing personal information like this - as soon as a site is visible on the internet there are bots trying all conceivable urls on it and scraping for information. If you look in your logs for any server you'll find all sorts of php,aspx etc urls as bots try to find vulnerabilities, no matter what you're running. I'm sure there'll be some Rails scrapers out there too though perhaps they're not too common yet.
There are probably a lot of other holes if they left the user security so wide open.
It shouldn't be something that slips through testing. If you aren't doing that from the start, something is seriously wrong with how you're building out your application.
It's a problem of competing claims -- you want to keep the world safe so end users are protected, and are willing to use new (secure) services, but you also want to avoid discouraging developers (either these guys, or others who see how they're being ragged on and choose not to develop something on their own).
It's not a fundamental flaw in the application, just an admin interface error. Yes, they should have known to test, but I reserve the nuclear hate for willfulness, since hate and vitriol is sometimes in short supply.
Just yesterday we had someone publish a "securely delete your email" application. 'tptacek found problems in it immediately[1], but he didn't call the guy incompetent or an idiot or "never trust anything he does again." There was no attempt to shame.
I see the more experienced people around here have a lot more sympathy for these guys. If you've done a lot, you've also had some public mistakes. You grow empathy.
I do find the company's follow-up offensive. Hopefully they will learn from that, as well.
I believe this was one of those cases. The founders of this application were told in the original thread that there were security issues. They didn't respond to the issue and continued allowing users to signup.
Their immediate response should have been to shut down the application with a maintenance page. Their response was instead to tell users to delete their accounts[1].
The other factor here is that because of the type of application users were likely to upload private and sensitive information. This wasn't a simple todo application where users would test it out with fake data, it is a backup application.
The combination of poor initial response, the sensitivity of the data being used and the popularity of the application (being at the top of HN, all over twitter etc.) would lead me to make the exact decision what this blogger did. It was important to notify all users asap that there are problems here, so that they could act on it.
Edit: didn't you do something similar with the Diaspora launch? I think that was another example where it was important to get the vulnerability information out since that first release was popular, users were uploading sensitive information and it was going to take some work to secure the app.
FTR, I don't think that the gap between saying there is a security vulnerability and describing it is very large, especially when the audience contains capable penetration testers.
I know it's hard for people behind this company and they probably invested a lot of time and love to build this product. But we can't just let that pass, for the sake of people that'll use that service next (I mean, with something that basic missed, what next ?).
Please, just get back to learn creating web applications, and see you in a few months for a great product ! (because, yes, the idea was interesting)
With things like that, especially when revealing it could lead to people using the vulnerability maliciously I don't think it is ethical to release details of it, unless they don't make any indication they are going to fix it.
It is bothersome when they don't even thank you for bringing it to their attention however.
There is no such thing as irresponsible disclosure, therefore there can be no such thing as responsible disclosure.
For a vulnerability as obvious as this, it's a fair bet that bad guys will notice immediately. "Responsible disclosure" is great when you've discovered something tricky, but it's irresponsible when anyone else can notice as easily as you can.
Remember that the term "responsible" is about responsibility to the users, not to the developers. If publicizing a vulnerability would leak it to bad guys who don't have it, the responsible thing to do is not to leak it. If the bad guys already have it, the responsible thing to do is to tell the public. (After all, disclosure is about whether to tell the public, not whether to tell the developers.)
That wasn't merely a "security vulnerability". It was also a demonstration that the people running the business have absolutely no idea what they are doing when it comes to security, privacy, or testing and release processes. (Actually, there is an alternative explanation, which is even worse: they knew and didn't care. I prefer to assume naivety rather than malice.)
Unfortunately, the only sensible action when faced with a business like this is to run away and not look back for a very long time, except perhaps to check who the people responsible were so you can avoid anything else they work on in the near future as well.
Fwiw back in 1996 or 97 the UPS website did the same thing. By altering the tracking number you could see somewhat complete information on someone else's shipment. Since the tracking numbers ran in sequence from the shippers log books giving one tracking number from a competitor you could see all their customers. (To get that all you had to do was place a single order so they shipped to you. Although I guess it wouldn't have been even easier to social engineer someone to simply give you any tracking number and save that step.)
I agree with silhouette. These guys scaffolded a rails project and and then slapped bootstrap on it. You can't trust an MVP this extreme.
The fact that the results are so different is irrelevant - the attack vector was essentially the same.
Also, even with the complete lack of security on this site, it should still not be possible to take any action on the victim's AWS account. IAM has read-only roles for this exact reason - hopefully no-one was negligent enough to post their master AWS key/secret in to this or any other third-party site.
Obviously if there is a known attack vector, you would fix any similar issues everywhere, but not all code is the same.
In practical terms, of course the nature of data that is disclosed is relevant. AWS keys are incredibly valuable, and should be treated as such.
Package weight
Shipping date
Who signed for it (last name)
Where package was left
Town delivered to
When delivered
And some other nominal info.
In the old days you saw exactly who the shipper was and detailed info on the recipient and recipients address. There was probably other info but what I've listed is what I remember. I remember thinking at the time that it would be valuable and contain exactly what a competitive company would need to gather a list of potential customers.
I'm pretty sure a lot of successful startups were started by people who had "absolutely no idea what they were doing". Give them a break...
The big fuck-up is when they told anyone with key problems to contact the guy who found the issue. That's why we should consider them unprofessional. The security holes were accidents. The blamestorm was deliberate.
I go one step further. I refuse to give information that provides more access to a business than they need to have or that can even affect any other service I receive from anywhere else.
Here, I'd like to give them a key that works only with glacier vaults that they have created, and nothing else. If this isn't possible, then I'll go without.
It's absolutely true that you can use ORM and scaffolding patterns in a totally secure way. But the problem is that the defaults are insecure -- every table can be accessed, every record is available, every field can be edited, and the URLs for doing so are (deliberately) easily guessable.
One of the simplest and most fundamental rules of effective security is to close everything down by default and only open things up as required, after careful consideration. Scaffolding breaks that rule.
http://pragprog.com/book/rails4/agile-web-development-with-r...
This is really an argument for building authentication and authorization into every app, rather than against scaffolding/ORMs.
As rails doesn't have auth (of both kinds) built in, it doesn't really matter if they offer scaffolding or not - any editing url you make is going to be completely without protection unless you add it. The only thing you'd be adding by not having guessable urls without authentication/authorization is security through obscurity.
So IMHO the lack of auth is really the issue here (and the thing that breaks the rule in your final sentence), rather than the guessable urls.
Which is why my Rails authorization library takes a whitelisting approach.
Well, scaffolds are not suppose to totally avoid coding. They try to provide what you may write again and again, but you're supposed to take that as a basis, not a final product.
EclipseLink ORM http://wiki.eclipse.org/EclipseLink/Examples/JPA/Multitenant
PostgreSQL Veil add-on: http://veil.projects.postgresql.org/curdocs/index.html
Multi-tenant Data and MySQL (trigger + view method): http://blog.empowercampaigns.com/post/1044240481/multi-tenan...
MSDN Multi-Tenant Data Architecture: http://msdn.microsoft.com/en-us/library/aa479086.aspx
The data leakage obviously overshadows it, but I can't think of a site that wouldn't be a better "fit" for SSL encryption than an app like this, aside from banking/government sites.
SSL might have been a "nice-to-have" back in the day when there were real arguments to be made against it (mostly performance-related), but even those don't really apply to a "pet project" made by "two nerds" (smeagol's words, not mine.) And for an app like this, I think it's critical.
Just my two cents.
It looks like this was just the default Rails resource scaffolding.
Only if you're concerned that you can't tell who is "relatively senior." In this case, your judgement was unfortunately wrong.
I have to be honest - that doesn't demonstrate a high level of trust at all these days. It's sad, but true. Plus, if you say they're "ex-WePay," I assume they were just everyday developers for WePay, not critical resources.
If you let them know and they ignored you, then I understand that you'd want to write an article and spread it around. It's important that customers know when a company doesn't value their security. At that point, the proper way for them to handle it is to quietly fix it, and then let all their affected customers know so they have a chance to change their security settings.
However, if you didn't give them a bit of time first, then you are doing more damage than good to them--and their customers.
How this happened is what I want to know too.
As people have mentioned, rails doesn't have it built in. I've used gems to provide it since I don't trust myself to write good enough security algorithms (and really, why reinvent the wheel if I don't have to).
In .net we can use the asp.net membership. But you've always got to have that authorization part, which I think can get forgotten about unless you've got a system under you belt or something/someone to crib from.
Sometimes you just don't think, and sometimes it becomes very public.
You can even create your own custom filters.
http://www.youtube.com/watch?v=BsxUsyMSGeA
Just letting you know. :)
However, I do think that authentication is where people may believe they can stop, forgetting or maybe not understanding, that authentication really doesn't do much, without an authorization system.
I'd also encapsulate use of any user-provided sensitive data in an API then called by your service, and put some logic within the API (because I don't want a random web UI screwup to dump everything for everyone) -- rate limits, etc.
Any site that employs the pattern of specifying user account routes using the user's primary key in the URL needs to implement authorization. This site clearly skipped that step.
To me, this looks like the stereotypical bare-bones rails deployment by a newbie.
That in itself is not a security problem, but having no access control obviously is.
They therefore created a default resource structure (probably using the rails scaffold generator command) which includes the id in the URL on all member routes, including the route for the edit action.
Maybe if there was a service promised but not rendered, could you place full blame on the developer(s).
On one hand we all want to move quickly, get users, add new features, etc etc.
On the other, security issues like this are just so vital that nothing else really matter if your data is not secure. It's especially true for a BACKUP SERVICE that promises ridiculous stuff like "99.999999999%" uptime on the frontpage.
honestly, this was all accidental. it was a pet project we started to toy with Glacier and a week later i accidentally hit the Like button sending a ping to my friends on FB. bless my friends for being so influential i guess. shame on us for using Rails carelessly.
if you have any experience with startups, you'll know that 99% of the things you launch go nowhere--this project was no different. we honestly thought our site was of absolutely no consequence. we're truly thankful so many people found it useful, but trust me we're sorry there was a hole.
however, just to be clear:
- about 20 accounts were exposed, including me and my buddy - i emailed all of them, and wiped out the credentials - they quickly responded (i saw the updates come in)
thankfully, AWS is designed for such situations. with a few clicks, people deactivated their credentials (both IAM and main account) and regenerated new credentials. the fact that all the early signups were techies who know their way around AWS really saved us.
one more thing: the correct quote is:
"Glacier is built for durability of 99.999999999%"
also: i agree with ryan--don't trust 10-minute old startups :-)
You have a long way to go in my mind, in terms of fixing the initial response. You probably have help now, which is great, but your initial kneejerk demonstrates underlying trouble to me which you need to fix.
You're in a tough spot, too, because you can't delete those godawful comments without looking suspicious.
I'd like you to apologize not only for the disclosure, but also to the reporter for how you treated him in the other thread. The entire other thread of your responses is disgusting, and you don't get to write it off because of your gender, quantity, or employment status. Own your comments and stop excusing them with that bullshit line.
I have to admit that I would also be pleased if your service disappeared until you're working with somebody who has a little more experience with secure Web applications; this mistake betrays your experience. Since we all started somewhere, though, I can only hope you fix this on your own.
Also, can you explain what "Glacier is built for durability of 99.999999999%" actually means, if not uptime?
If I got my math right, this means that they expect to lose on average about 10 bytes per stored terabyte per year. (Of course these losses, should they occur, would probably be not uniformely distributed).
"Pushing it to a public server" is really minor. Mozilla had this issue, too, when they had a new filename technically available on a server and someone jumped the gun and told the whole world that the new version was ready. Well, it wasn't. A bunch of kids whined that it was all Mozilla's fault for having a file available on their public server, but while it's arguable that a service that is reachable by URL has no expectation of privacy, it's a hell of a lot harder to argue that having a service reachable by URL implies a warranty that it is safe to use.
Friends in the 90's would run telnet and web servers with "Username:" "Password:" "Credit Card Number:" prompts. It was funny to watch that some people would type in apparently real data, although we never verified.
Cancan is great way to make sure that you can only read or edit your own records in the database with Rails.
It's also interesting that the aws key/secret are "masked" on the page, but you can just visit http://www.iceboxpro.com/users/12.json and get the formatted json representation with no masking.
If it's not your resource, it's like it doesn't exist for you.
The truth is that I don't remember working on a single codebase that didn't have some eventually discovered vulnerability in auth(entication|orization). When I eventually do comb through controllers and find easily exploited access-control violations, I've often been met with responses similar to the behaviour of the developers at Icebox.
Rails does and will continue to protect you from a lot of mistakes, but nothing is going to help long term unless you know what words like authentication, access control and session management mean.
If you're a professional web developer and you care about your users then please buy and have a read through The Web Application Hackers Handbook[1]. Every page is dripping with easily exploitable attacks you didn't think of. That last app you built is almost definitely vulnerable to a handful of them.
[1] http://www.amazon.com/gp/product/1118026470?ie=UTF8&tag=...
if you've used that service, the information you entered was publicly visible (key to access aws, etc) (the thread linked above says it has now been patched).
[i don't understand why, but when i access the link for this thread i get the gzipped page as a download; linux + chrome 22; firefox displays what appears to be gzipped data; wget saves the gzipped data as index.html; same behaviour for chrome on opensuse and ubuntu; windows 7 + ie9 (in a vm) shows the gzipped data in notebook; is no-one else seeing this?!]
[update: fixed now - it looks like it wasn't changing the content type]
In this case you have a strong point. Good work finding the security issue and reporting it.
It would have been more responsible to privately notify the owner of the site rather than karma whoring a blog post to top of HN.
How did you not notice that?
Tell me that blog post didn't read like a security exploit announcement...
Anyways, as discussed, it never should've happened in the first place, but it sounds like from the comment threads that it was likely Framework related, so.... yeah, still prettty bad.
I don't know the rails solution, but a quick-and-dirty solution in other frameworks is to use a decorator on your controller/views that does something like:
if request.session.userId == action.userId:
pass
else:
return SecurityExceptionResult
With that said, user-identity does not belong in a URL. If you just did /user/edit (we assume all operations are performed on the logged in user) and then moved your security validation down a level to verify that session.userId == model.record.userId you'd be much better off.
My point was the amount of time required to prevent security holes like the ones outlined in the link are minimal - preventing them isn't going to stand in the way of an engineer implementing other features.
What strange times we live in.