A single developer dropped AWS costs by 90%, then disappeared | Hacker News Reader

A single developer dropped AWS costs by 90%, then disappeared

medium.com

146 pointsdevelatio61 comments

Reminds of a developer I hired through Fiverr that claimed he could achieve 99/100 Google pagespeed score for any Shopify website (and improve overall load times of Shopify). He implemented his changes, and sure enough it scored 99/100. However, the site still seemed to be running slowly when I tested it. I dug into what code he had actually changed, and there were only a few lines of JavaScript injected into the site. The JS was obfuscated, after de-obfuscating it I realized all his code did was prevent any other JavaScript from being loaded if it detected the user-agent was coming from Google pagespeed :)

After some back and forth with Fiverr I was able to get refunded, but I'm sure the thousands of other customers that gave him 5 star reviews have no idea.

If the service was advertised as "improve your page load times" then I can see how that's misleading, but if it was to get a 99/100 page speed score as a kind of "dumb" SEO hack then maybe you got what you paid for?

It was a service advertised to improve your Shopify load times with a number of "optimizations" not just pagespeed score. Updated the original comment to clarify.

The load time is certainly way better with all the js disabled

To be fair, the value in getting a 99/100 pagespeed is SEO optimization, so sites rank higher on Google. The point is getting more traffic to the site, not a better user experience.

But doesn't Google penalize you for user agent based cheating?

Yes. Serving content crafted for spiders specifically is a blacklist waiting to happen.

Good thing you were able to get that refunded.

Ah! The good old VW diesel dupe solution.

> A few months later costs started to rise again. Each time a GCP free-tier account did not work, data was sent back to the old AWS system.

I like this part. He may have run a total scam, but he still wrote good code, dammit.

The company also still saved money in the long run, just not as much as they had hoped initially.

The money the company saved would have paled in comparison to legal costs had Google or some government authority decided to take action against them for buying and using stolen accounts. And no "this random developer (who we can't find) did it" isn't a valid excuse. The company is still fully liable for their code.

Does google have no liability for offering free accounts? Seems some abuse is to be expected when you open yourself like that. They failed to validate the owner of those free accounts.

No they don't and they shouldn't.

And still saved them about half a million dollars (6*$90k - $50k).

IMO this is what allows him to escape guilt free. Although his solution didn’t last, it lasted long enough that the money saved was far greater than if they had not paid him $50k. This could be a good hack for a startup that needs to temporarily reduce its cost for some short period of time.

Do this as quickly as possible for a a dozen or so clients and you could build a decent nest egg in a short period of time.

Wonder what the lawyer costs would be should google go after the company that paid him to do the fraud for them?

Sorry, why are we believing this story? The technical details make little sense; e.g. moving a ml processing pipeline to a different cloud just by "editing a Chrome extension" doesn't seem very plausible.

It isn't even internally consistent, the hallmark of a sloppy fabrication. At first the accounts were sourced from the darknet, then they were created by the Chrome extension.

Creating a million Google accounts is pretty hard. And buying so many accounts from the markets would probably cost his whole $50k price.

Also, it was a million accounts and the savings were about 500k. Each account spent 50 cents worth of compute out of the $300 free trial? I assume you need to give a credit card number for the free trial. I wonder how many of those were needed for this to work...

Your skepticism is warranted, but just to note:

> At first the accounts were sourced from the darknet, then they were created by the Chrome extension

The article says they bought Google Accounts on darknet and _created GCP accounts_ from the chrome extension. Presumably "GCP Account" here means a new GCP project/billing account with free trial or using the always free usage. So it's vague and difficult to believe but not actually contradictory.

It looks like the story is just something the author of this post pulled from a Twitter thread (linked at the top). The thread seems (I'm relying on Google Translate since it's in Ukrainian/Russian), to just say there was a big list of a boatload of email addresses sourced from the person's previous work, and it pulled from those to keep registering new Google accounts (or maybe log into them as there's mentions of passwords being included, and simply sign up for GCP on them).

There's also nothing about AWS vs. Google in the original thread I see, nor specific numbers on numbers of accounts, amount saved (tens/hundreds of thousands a month is as specific as it goes), etc.

Skepticism is warranted but the original tweets weren’t SEO-d down and seem more believable. This one makes it sound like this was a list of valid Google accounts which could be signed up for GCP:

https://twitter.com/xanf_ua/status/1608121336190357504?s=20&...

That makes more plausible: start with a list of Google accounts, try to use it with GCP, and cycle accounts until you find one which hasn’t exhausted the free tier. It would be very easy to believe that the 1M account number was winnowed down rapidly by the ones which had changed passwords, tripped some kind of suspicious activity alert, etc. or been used for anything else if that wasn’t the only copy of that list in existence.

Last time I've used GCP, the trial required a valid credit card that wasn't used for another trial before. They didn't take preloaded card either. So yes, this story is definitely... hard to believe.

I'm almost annoyed the SaaS company in question profited from this for those 6 months before their costs started going back up again. Any IT Director who bought into this snake oil, allowed the person access to their codebase, and accepted their changes without review should also be fired.

The "IT director" was probably too busy going to zoom meetings, fighting fires, etc. to do those sorts of reviews. He should've delegated it to a lead dev, though.

They don’t have devs as I understood it. They relied on external developers for their software.

If something seems unbelievable, it probably is. How can anyone not question how a 90% reduction was possible without proof?

That said it looks like the company did save money for a few months so the 50 grand was worth it although hard to maintain indefinitely and of course against ToSes and morally questionable.

"probably" but not always. I have worked on a few projects that saved 70-90% of costs for AWS -- but these all were starting from a position of auto scaling out a ton of ec2 (on demand full cost instances) to batch process huge amounts of data. In both scenarios by optimizing the batching to align tasks to pack as much as possible into (just short of hour long) hour long windows and converting to a (wide list of ec2 types to avoid contention lockout) spot instances there was easy fodder for the improvement.

But yeah -- no one should see their bill change by that much without understanding exactly what was done to trigger it. Even in the best cases, there are always tradeoffs that must be known and accepted.

Agreed with both you and parent commenter - the story comes across as almost apocryphal; a warning both that if it's too good to seem true, it probably is, and that one must always understand the work done and the tradeoffs involved (IE: don't fall asleep at the wheel)

To be fair, a lot of cloud deployments are not optimized for cost, and it's easy to rack up big bills without really needing to.

To be fair, as an IT director, when you see this sort of reduction you should dig deeper and understand where the cost savings are coming from.

Reducing 90% cloud bill is not out of the ordinary.

Reducing cloud costs, especially if the dev team does not have anyone experienced enough to focus on this, is often possible on massive scales. More so if you have free reign to optimize/rewrite stuff as you see fit. While not 90%, I've reduced cost by 83% in one case while increasing the ability to serve during peak times also.

90% savings is very easy if the system was built by a beginner. I helped a friend once who was starting and reduced his query by 99% of time, and thus quite a chunk of money, by a couple of simple changes (IIRC avoiding a JOIN he was making that he didn't actually need and adding an index).

I mean, you can go from poorly-sized EC2 instance with autoscalinggroups constantly interrogating remote databases to full serverless with databases hosted on AWS, and 90% is an easy target.

Dupe of https://news.ycombinator.com/item?id=34281802

Yeah, I remember seeing the original with the fictional disclaimer. Wonder why it was edited.

before/after (you can find a snapshot on archive.org:jan 6, 2023)

> NB: Fictional story, inspired and translated from this source (https://twitter.com/xanf_ua/status/1608121311078776832)

> NB: source (https://twitter.com/xanf_ua/status/1608121311078776832)

I was planning to drop a long-winded comment on that submission about how this kind of abuse of free tier services is why we can't have nice things but decided not to.

Because the story was true...

Wonder if stealing a companies slogan ““We Shall Build Good Ships Here; At A Profit If We Can, At A Loss If We Must, But Always Good Ships.”” — Collis Potter Huntington is considered the same thing?

Pretty clever scam. He had a huge list of accounts, then set up a script that took advantage of the free introductory compute AWS gives to new users. When the script ran out of the accounts to cycle through, it fell back to the old system.

How can you consider it a scam though? They paid 50k and saved hundreds of thousands according to this story (90+k reduced by 90% for 6 month.)

Google is the only one that got scammed as far as I can tell.

It's a scam because (1) they did not promise a cost reduction for just a few months, and (2) what they did was illegal and/or against the terms of service of the company's cloud provider. Just like how selling you a perfect working car for a great price – which just happens to be stolen – would be a scam.

They thought they were getting optimizations to their AWS infrastructure, but were actually using stolen resources from GCP.

They should've bought a few more million accounts on Fiverr and updated the yaml file.

This is why you need strict code reviews, and leaders who actually understand the technology. Any competent organization would have caught this.

So you need a developer to review a developer. Who's reviewing the developer's developer?

Another developer

"developers, developers, developers, developers!"

A good story, and likely untrue although still possible.

I've helped clients save huge chunks of cash over the years from AWS because the previous developer/team paid little attention to costs.

Reserved instances, alternative routing of data, cheaper S3 tiers, etc. soon add up significant savings.

Ah, the original poster in Ukranian* (_ua) just deleted the thread.

Original Twitter thread in Russian is here: https://mobile.twitter.com/xanf_ua/status/160812131107877683...

But it's illegal. It's abusing the system.

This ticks me off every time i see it. Neither Amazon, nor Google or any other tech company are allowed to make laws. It's a breach of the terms of service maybe, and maybe illegal, but whatever the terms of service say is not a law and said terms of service could be themselves illegal.

They have enough power, please stop attributing them even more by your choice of terms.

It is most definitely illegal to buy and use stolen credentials from darknet markets (which is what the author hypothesizes happened in this case). It is also illegal to misrepresent the service you are providing to a customer.

It's likely wire fraud

free credits is not money. FAANG can't just create money by saying we give you 100k credits

Wire fraud doesn't require that you're stealing money:

> Whoever, having devised or intending to devise any scheme or artifice to defraud, or for obtaining money or *property* by means of false or fraudulent pretenses ...

(18 USC 1343)

Property here includes stealing services of value (in fact it even includes intangibles such as confidential info or goodwill, see Carpenter v. United States 484 US 19).

It is not that they are creating new laws, it's that the dev committed theft of services and fraud (existing well established laws) as he utilized the system with intent to receive free services that were directly forbidden in the grant from google.

Being against the terms of service is not illegal.

They are different things.

Google might have a case for fraud, but that would be unlikely to move forward given the circumstances.

It’s against the terms of service, and a bad idea because you’ll get banned and lose all the data in the free accounts, but is it illegal?

Not a lawyer. It's likely fraud. (Doesn't matter because the story is fictional anyway)

Technically, breaking a contract is not illegal. But a broken contract is evidence that can be used to sue someone for a crime.

If our contract was "you pay me $50 every month for a year and I give you service in return", and you don't pay me, I can sue you for fraud and use the contract as evidence.

But if our contract was "you must say 'Hail, Emperor!' every time you see me", and you break that contract, there is no law that enforces that.

Reply on news.ycombinator.com