$36k Google App Engine RCE
sites.google.com
sites.google.com
To find the bug is impressive. To write about it so well is truly exceptional.
Who do you sell it to? I assume your answer will involve putting it up on some darknet version of Craigslist. That's fine, but then tell me: who's paying for it? What price do they assign to it? For instance: if you think you can sell it for $50k, who's paying that, and for what purpose?
Finally, what are the steps you take to safely complete the transaction?
(This is intended only to clarify arguments about the market for vulnerabilities like these, and not to suggest that the finding and the writeup aren't excellent, which they sure appear to be.)
Notice also that with just a couple exceptions, RCEs in extremely widespread serverside web components are valued at $10k (if you believe their price list; I'm skeptical of it). Those are vulnerabilities that all have half-lives after patches are issued --- that's a ceiling for what anything like this could be worth.
Second, Zerodium isn't "the black market".
In the scenario you described, without any other contacts and/or experience with transactions like this, I would approach an exploit broker. As for the payout - I assumed that any RCE vulnerability that qualifies for Google's highest bounty is likely to fetch a higher price elsewhere.
My experience with the field is limited and considering yours, if you are suggesting that this particular exploit would not fetch a significantly higher price, I shall stand corrected.
I'm not quite sure what you're getting at here. If you're trying to point out that I haven't done my homework on this and that I don't have a sufficiently specific/workable plan how to approach it - that is accurate. I don't have exploits to sell.
In my previous comment I already stated the assumption that I made, if you feel it's incorrect, which clearly you do, feel free to correct/fill in whatever you think is missing. I'm not getting into a debate about something with which I don't have in-depth experience with someone who does.
I think a lot of HN'ers believe that there's a market for high-severity bugs of any ilk, when, in reality, there's really only a liquid market for a pretty specific subset of those bugs.
As for the black market price - I don't consider my security background sufficient for my guess to be anywhere near educated enough, so I'm bowing out.
And not to spoil the game, but the subset of vulns that fetch good money has only narrowed in the last years as exploit mitigation has improved. The true unicorn 0days of yesteryear are almost always multiple hard earned bugs these days. Bugs in one vendor's project, even Google, it is cool they have such a high end reward, let alone 36k. Unless you crossed a line and exfiltrated data (high risk), I can't imagine getting this much money anywhere else.
Amusingly, 36k does look very similar to ~3wks of boutique infosec consulting, though, so for Google the price while generous probably makes sense.
Known security person buys this in order to turn it in themselves for a higher bounty and increased rep.
I guess Zerodium has enough reputation that you can be reasonably sure they'll fork over the cash after you show them your bug. Building that kind of reputation on the darknet with cryptocurrency isn't going to be easy. If there was anyone like that out there, we'd already know about them.
BTW -- not to fan out too hard, but I put together the pieces recently and realized in addition to Latacora you worked on Cryptopals and Microcorruption. As someone interested in exactly those areas with little knowledge of how to break in, thank you! I've found both great learning resources (although I'm embarrassingly early in the crypto challenges still..)
In the GAE case, Google can patch all deployments of their service on the same day, which is an "instantaneous fix".
If it were an RCE with a webserver, OS or some other popular software, that can't be patched "instantaneously" "worldwide". Each entity who maintains their own (i.e. nginx, Drupal, Redis, etc.) instance needs to be alerted to the patch and then apply it. That's an incredibly long tail.
Or else, why do buyers want clients?
The hypothesis I like to come back to --- I'm pretty sure it's true --- is that vulnerabilities have value on the black market only if they fit into an existing business model, such that they can be dropped in and immediately be used to make money. People have to already be using some other vulnerability to do the exact same thing, and reliably making money with it.
People on HN like to tell stories about how a master criminal could make money with everything from Facebook CSRFs to serverside RCEs. But none of those kinds of exploits support current ongoing business concerns; they're all one-of-a-kind. Nobody buys a vulnerability speculatively to see if they might make a go of it --- they especially don't do that for a vulnerability that could be extinguished universally in moments by Google or Facebook's security team.
I could see an argument here towards the market price of the information -- breaching a company in the above scenario might get you zilch. Breaking an iOS device of an individual of interest gets you a lot of value.
Circling back though -- you're saying that this model isn't one in use. Do you agree with my real-life counterpoint? If so, why wouldn't there be a market? Or, is the market there and the low payouts from the likes of Zerodium reflect the actual low value of the product (and by extension, business model)?
There are, of course, real brokers who will buy zero-day vulnerabilities for use by national IC's and LEO's. Their names don't get around like Zerodium's --- Zerodium sponsors conferences --- but they're not hard to find.
If you've got the kind of bug that these firms buy --- essentially, clientside RCE in hugely popular platforms --- you can probably do better selling to them than you can by collecting bounties from the vendors directly. It takes some moral flexibility, though, since really what you're doing is profiting from other people's exposure, and, especially with mobile clientside RCE, what you're really really doing is getting dissidents in Western-friendly dictatorships imprisoned. But you can do that.
But none of these firms (that I know) buy one-off vulnerabilities like a GCE RCE. All the vulnerabilities with high market values have half-lives, which is to say that even after they're patched, it will take weeks, months, or sometimes even years to see them eradicated, which gives them the residual value that props up their market price. In contrast to that, a GCE RCE that was actually exploited would be detected pretty quickly, and shut down with finality in a matter of hours.
And, really, dictators don't need help imprisoning dissidents -- it's sort of in the jobspec. It's not likely that a vuln would cause this to happen unless it was in systems created and secured specifically for sedition, in which case a researcher should be considerate of the potential damage.
I would find it morally offensive but I'm a naive country bumpkin I guess. I have pointed out something that was mispriced rather than profit from their error. It cost me something like ~$50-100 extra but I slept better.
The only way you could sell it is if you already had connections directly to people who were known to need an exploit for this service and had a channel to approach them privately, IMO.
For someone in the author's situation, I think he got a very good deal and almost certainly made some great contacts that he will be happy to have in the future.
(That's a long way of saying that when I run that particular thought experiment, in the context of a vulnerability where discovery provides a certain amount of disclosure should anyone go back and check logs, I have a hard time seeing a more lucrative black market. I think the math would be different in less centralized cases.)
But back to your question. If I remember correctly, the big companies started giving higher bounties because they noticed that some of the bugs are being sold on the dark market. Some of these bugs are certainly not unmarktable but the companies have to keep the promise.
Also I'm not an expert on these things. Maybe there is an entity buying whatever bug you can find and giving generous money. Think governmental institution who might be interested with whatever backdoor they can get.
The $2k limb is shaky because I guess in theory you could buy a GCE RCE for $2k and flip it to Google for their bounty payout, which will probably be at least $3,133.70.
He'll have to pay taxes on it though (if he has no other income it won't be that bad, maybe 20%).
> a student from Uruguay who worked with Sugar Labs. Sugar Labs is the organization behind Sugar, the operating system for the [One Laptop per Child] XO-1 which the Uruguayan government has distributed to public primary schools. The XO-1 was Ezequiel’s first computer.
> Ezequiel’s curiosity in computer science was piqued when a technician came to his school to solve a simple bug that was affecting most XO’s. The technician used the command line which, up to that point, Ezequiel thought was useless. Realizing that the command line offered him a lot of power, Ezequiel began his exploration.
Wouldn't the investigation lead to him? He noted he did not know, at the time, that it was an RCE. So he would need to research further (where he might trigger an alarm). He also noted he utilized staging environment, which he had access to due to previous found vulnerabilities (so Google had his personal details).
So, now imagine he found something and he had sold it. If it was used, Google would do a thorough investigation to find if given vulnerability was abused in the past. And they find this guy using it exploratory and nothing else. It's not hard to put 1+1 together after that.
While the idea of finding an exploit and selling it for hundreds of thousands of dollars on black market sounds exciting, it wouldn't be so easy in this case.
See also tptacek's comments in this thread.
Someone could say that he could have gotten even more money by selling his findings in the black market, very difficult but doable. However, as someone who understands how studying computer science in a 3rd-world country is, getting USD +36k in a legal way and from a company that is considered one of the best in the industry, it must have felt very good to get that mail.
Congratulations, and keep the good work.
The skill, I guess, comes in piecing it all together and consistently making good guesses.
1st world: US, UK, West Germany, essentially western countries
2nd world: Soviets, East Germany, communist countries
3rd world: everyone who doesn’t fit the Cold War theater.
Either way, a GDP per capita of $15k isn’t considered “first” at whatever category you want to assign it by. Unfortunately there are no “1st world” countries in Latin America.
First world were countries allied with the US. Second world were countries allied with the USSR. Third world were neutral countries.
So Switzerland, Ireland, and Sweden are third world countries.
Really?! You do but you don't but you do want to be a stickler? ....
PS West Germany and East Germany are not separate countries. :)
I’m only mentioning this as people usually attribute it to only economic power.
“studying CS in 3rd world country 35k is a lot of money”
A "third-world country" was a country that wasn't aligned to either the Western countries (first world) or the Communist Bloc (second world).
These terms have become outdated, and "developing" countries is more commonly used now by, e.g., the press.
https://en.wikipedia.org/wiki/Developing_country#Country_lis...
Students shouldn't need loans. It's an aberration.
It's not fair on those who went directly into the workforce or became an apprentice that the university students get free money for living costs for 3 or 4 years.
The current system in New Zealand (simplified slightly) is that university is free, there are no fees, and you can get up to $180 per week for living costs as an interest free loan. I think it's a perfectly reasonable system. That amount isn't really enough to cover all your expenses unless you are especially frugal, so students still need to go out and get a job (or work in the summer), which I think is perfectly fair.
Obviously the upside is that you get a job where it is reasonable to expect that you'll be able to repay the loan in about 10 years, but that's something you worked for, and something an ambitious person in the workforce might achieve as well.
Anyway, I don't think you can simply say it's unfair to aid students in their living expenses while they're studying. Obviously they're usually having a great time studying, but they also don't own a car, or a house and instead live in tiny apartments sharing privacy with other students. They usually delay their family building for the entire span of their studies, and often even until they've paid of (most of) their debt. But yeah, have them work some extra in the summer while the workforce is driving their caravans to southern France...
“Please stop exploring this further, as it seems you could easily break something” has got to be the best reply one can receive to a bug bounty report.
There have been few stories on here of bounty hunters who've either been scared to go any further than they have, obviously giving the remote end a good excuse to pay less; or who've taken things slightly too far, generating mildly put-out emails and stepped-on toes (which can't bode too well for future interactions).
So this the best thing you can hear from that standpoint as well.
Straightforward communication seems to be a holy grail, which is understandable: besides needing full internal access to all the areas affected by the full scope of vulnerability (either directly or via intermediaries who are good communicators) so you can figure out when to hit the stop button, you also need to have the resources to delicately balance management risk and developer egos.
That sounds like a "fun" job...
---
I guess this demonstrates the fundamental requirement of security: the recipient is the one who needs the non-broken understanding of security, both a) properly in and of itself, and b) how it relates to the domain they're in.
There are a thousand public examples of where this has been gotten hopelessly wrong; my bookmarks are a mess, so here's the (somewhat poorly related) one I can remember how to find via Google (it's as bad as all the others, wouldn't mind being reminded and learning about new ones). This one is not the same kind of vulnerability (there's no need to coordinate pushing a stop button, it's a data breach), it's just an example of grossly misunderstood security and utterly broken internal communication.
- https://arstechnica.com/information-technology/2018/04/paner...
- https://news.ycombinator.com/item?id=16739753 / https://news.ycombinator.com/item?id=16741391 / https://news.ycombinator.com/item?id=16737583
How scary must that be for the Google team? You know you've messed up so badly and the person who is investigating is doing so blindly with no knowledge or accountability if he breaks something. Yikes.
Kudos to everyone for doing the right things. And great bounty- the average yearly income in Uruguay is $2000-$3000 USD per household. This guy just got awarded more than ten times that.
[1] https://www.ceicdata.com/en/indicator/uruguay/annual-househo...
It was really fun to read the forums and see how, day by day, they managed to get closer. Since it wasn’t really crucial IP to begin with, we were rooting for the little guys to see how close they would get, secure in the knowledge that our algorithm was solid. :-)
The final exploit that granted them access was due to a supplier who replaced an earlier validated random generator with something not quite as random, which enabled replay attacks.
I’m curious why there was no auth required for his calls.
* Make the right requests to one endpoint he found and retrieve company financials, number of hits to every google service, the name of every application running in every datacenter, etc.
* With the above two things, you know the location of services and every RPC endpoint on them, and all access control configs. You can take your sweet time to audit the 10's of millions of lines of code to find vulnerabilities and get to attack as an authenticated (albeit low privilege) user. A lot of stuff is open to all authenticated internal users.
* For example, you could take down any google service by quitting all the application servers at the same time by calling the right debugging RPC. You'd be caught obviously tho.
36k is not a small bounty for an RCE, but I feel like this is more critical to Google than the highest Android payout, for which they pay up to 200k for: https://www.google.com/about/appsecurity/android-rewards/
For example, with a decent remote android exploit, I could distribute a patched Google Play Services to all vulnerable handsets which disables updates and then listens to my own command and control infrastructure for further actions.
I can now hold the phones hostage and extort google for money to regain control of them.
But I think the same could be accomplished with the access he had, or worse, but would have taken a lot more work. He also would have needed to avoid detection too. His access sounds more troubling than the Aroura attacks they had years ago.
That said, it looks like in this case he would been able to turn on access to google3, through a special flag that is probably meant for internal GAE apps that work on source code. Presumably, the flag allows the app to use a GAE proxy that authenticates to Piper and provides a filesystem-like interface. It's not clear what would have happened at this point, because it is likely there exists a finer grained quota for source access than just all of GAE. These kinds of new RPC traffic would have been visible in Dapper and Census data, plus there are booby traps everywhere aka defense in depth.
The articles about Piper and co. have made it clear that it ran on top of Bigtable first and then Spanner, i.e. in the production network, spread over ten locations. There's file level auditing and access control. It would be very dumb of them not to have intrusion and anomaly detection in place.
And, you can't quit all application servers at the same time. These are distributed, self-healing, and highly-redundant -- meaning that there are thousands of copies of each service, and the system will bring up new copies to replace the ones you kill, without loosing service (though you could, potentially, affect quality of service).
Even the control mechanisms that allow all this magic to happen are run on the same platform, meaning that you can't affect the managers in this way either.
The sort of things you can do are: affect billing and reporting, find sensitive company data, and, potentially, execute code remotely (though that will probably be in a container, and not have access to much else).
Grabbing source code is problematic. There is more than one repo -- in an org this size there are probably millions. The code is huge, so downloading it will take forever, and then you'd have to read it. Finally, it's written in dozens of programming languages, some, like Golang, are unreadable to anyone but experts.
Google is using a monorepo.
How many would stop at "Eh I managed to fire requests to a hidden RPC service in google, but couldn't figure out how to make it do anything useful to qualify".
Put yourselves and your work/findings out there people!
I'm not sure what access that actually gives, but that email thread showing people getting +cc'd on the ticket speaks pretty obviously to the severity and magnitude of the problem.
(Am Googler.)
All that said an honest question: why would a company like Google not pay insane amounts of money for these kinds of bug finds? What would they pay their own people to find them? Seems like RCE on App Engine should be worth 100K+ and then some on top for giggles just because they can.
Obviously having a standard policy makes sense so that your community understands what to expect but as Google, what's your operational impact if you triple / quadruple vs. market value of the exploits?
My understanding of the pricing is that it's designed to make ethical behavior profitable enough that fewer people are tempted to sell the exploits on the black market, not necessarily to out-compete the black market entirely. I think this person's find is a great example - it's a resume-booster, a great experience, a very nice cash infusion, and helps, instead of hindering, their job prospects with future employers.
(From following this error: "Refused to display 'https://bughunter.withgoogle.com/0x0A?embed=1' in a frame because it set 'X-Frame-Options' to 'deny'.")
This would not improve his odds.
The Java 8, Node.js (just announced at I/O) and GCF environments use a new sandbox that should allow you to run any binary. Of course there are limits, and you have to pay to run things on App Engine past the free tier, but most things should run just fine.
The older GAE sandboxes didn't let you do this (and had a ton of other limitations as well)
Why's that? It's not retirement money. 400k (salary+stock) is one year of compensation for some Google engineers.
And very, very few Google engineers make that kind of money.
You'd be surprised. In Mountain View, everyone level 6 or above makes at least that amount, and most level 5s probably do as well. I'd guess that probably 20% of engineers are T5 or above, which is a ton of people when you multiply by tens of thousands of engineers.
Source: got a Google VRP reward.