Google refunded $200 because I missed 5 lines of Code | Hacker News Reader

Google refunded $200 because I missed 5 lines of Code

aswinmohanme.github.io

267 pointsaswinmohanme67 comments

I once worked on a platform where you could buy ringtones, wallpapers, jokes etc by SMS pre App Store and play store.

Every night the platform would kick off a process to claim money from the telco using some key which was tied to an SMS the user had sent.

That process was responsible for more than $1 million a month, but it was buggy as hell.

Half of the billing attempts would fail, the retry logic was buggy, processes would crash so you would just start them again without proper semantics to ensure you don’t bill people twice.

After a while there were so many retries outstanding that you couldn’t get through them overnight, so we would try in random orders and we would have the process running from 3 days ago whilst try close off today’s bills.

I’m sure people were billed incorrectly and a lot of money was left on the table from users who should have been billed. We never really heard a lot of noise about this either.

So I’m sympathetic to this process and glad that Apple and Google have improved the confidence around mobile commerce.

That's very interesting. I'm sure there are a lot of stories like this from back in the day - charging people has never been easier than it is now.

Not just back in the day, I have at least a couple of customers of which I know they are losing money because of billing issues.

Why? Because most accounting software is horrible and have minimal to no third party integration options. Many of them don't have an API, so the only option you're left with is manually (!) importing some XML structure every day. The logic is usually very limited so stuff like record mutations are often silently ignored during the import.

The accounting software world is evil. Deliberately engineered vendor lock-in, very limited third party integrations, ridiculous charges for 'custom' implementations, etc. I once build a migration tool for a well known accounting/ERP suite, and my partnership with them was immediately terminated due to 'breach of terms'.

Is this about SAP?

SAP has plenty of integration options.

There's a lot of banks in various countries whose standard “payment API” is uploading a CSV file with a list of account numbers to an FTPS or SFTP server nightly. And guess what happens if you upload the file twice by accident…

I would be more than happy if they just use FTPS/SFTP. I've had enough nightmare working with few that require proprietary Managed File Transfer software (aka MFT) e.g. HULFT, AS1/AS2/AS3 clients.

Google themselves stole $75 million from Adwords customers by not coding up a particular kind of refund and ignoring it for decades. It is likely a similar thing is happening with banned account balances.

https://www.businessinsider.com/google-emails-adtrader-lawsu...

https://www.searchenginejournal.com/adsense-lawsuit/248135/

That’s interesting! I’ve often remarked that it’s a giant competitive advantage for huge companies like Comcast to be able to net however many millions of dollars from billing “mistakes” that never get contested. On the one hand, I’m willing to concede they are mostly probably honestly mistakes and they only make up some tiny percentage of all their transactions, on the other hand, it probably adds up to a decent sized company’s total revenue every year that they basically get just because they’re so giant they can screw up a ton of transactions and plausibly claim it’s not fraud.

(Googler, opinions my own. I also do paymentsy things.)

Carrier Billing has definitely gotten better and easier. From the Google Play side of things, there is an API that is used to help with things (no public docs I can find sadly).

If you want to get a feeling of what it looks like, you can check out Standard Payments: https://developers.google.com/standard-payments/

Near the end of the article: These kinds of incidents make us wonder the real cost of Code and the more real cost of messing up

Imagine having an app about there ticking away and making pretty decent revenue. Little do you know there's a subtle error in business critical code (e.g. in-app purchases, ads, etc), reducing the revenue by over 50%. This is something I've seen first-hand and have become quite paranoid about. I suspect that it happens a lot. The reality is, no matter how many times one looks at their own code, there is no substitute for continuous testing of typical use cases in a production environment, ensuring that business critical code functions as expected. It's a continious process, because even if the code works properly at the beginng and isn't modified, the underlying OS or API's it interacts with can change and introduce unexpected behaviour.

I suspect this happens quite often.

There are ways to mitigate this: 1. Instrument, monitor, alert 2. Experiment, and convert canaries to experiments.

This is why software asks you for usage data

Add instrumentation. At some point in the code, the data before the bug won't line up with the data after. Watch the metrics and make sure they make sense. Validate the theories your have about what you think users are doing.

RE: Indie Devs shouldn't short on QA

I've worked for a company that went bankrupt over some accounting system they'd built and used themselves. It contained a subtle bug that made them think they had more money than they had.

It's really vital to thoroughly test and verify any financial code. Any code, but financial doubly so. And do it again in production. Even small errors can really add up over time.

Yes, testing is important but also sanity checking and in-flight checks as well

Why are you getting declined transactions? Why are you getting chargebacks? Are we getting one transaction per sale?

> In order for Google Play to ensure a consistent purchase experience regardless of where the user purchases your product, you must acknowledge all purchases received through the Google Play Billing Library as soon as possible after granting entitlement to the user.

Can someone who understands this domain better explain why this exists?

A superficial reading of the text triggers thoughts of "fraud prevention", but the code the author uses to fix it is clearly one a fraudulent app can also easily include.

Other thoughts about preventing poor user experiences such as dark patterns to trigger accidental purchases, or bait-and-switch functionality with Fremium apps also don't seem to be prevented by this.

So what's going on?

My guess would be to prevent lost delivery of the "payment received" notification from impacting the purchaser.

So Google knows that you know that they paid, would be my guess.

I was confused too because it sounded like the developer had to log in and click something to acknowledge the purchases. But it’s the app that needs to acknowledge that it has unlocked its features for the user. Makes sense to me now.

Play Billing 2.0 added the ability to purchase in-app items from outside of the application.

This ACK confirms that the application properly processed the notification and delivered the item to the user. Otherwise, Play assumes that the purchase was lost in the ether sometime before fulfillment (e.g. bug, the app crashed, etc.).

I assume that part of it is a psychologically-motivated desire to maintain a tight reward loop for Play purchasers, with no payment-processing delays.

How do you get that out of previous comments

Otherwise the item could only be available once purchase was confirmed.

How could it be available if the application didn't ACK that it received the purchase order and granted the item?

From the post, it sounds like the users receive the purchased item immediately but the app developer needs to confirm that the charge went through properly. Otherwise the funds are returned but the user keeps the purchased item.

Without acknowledging the purchase the money was refunded back to the people who purchased it. And the best thing was the dark theme would be activated even if the user got the money refunded.

This seems like something that static analysis can correct before the new app is even allowed in the App Store. If you accept a purchase but have no calls the the acknowledgement APIs then bounce with a message. Hell, it could even be a compiler warning. This seems squarely Google’s issue and the refund should be returned.

Yeah a static analysis would have been awesome. Now the refund amount has gotten to over $300 because of the time users tke to update their apps.

Or even an different API design. Eg a @noignore return value that guides/requires the dev to acknowledgement.

Static analysis aside, as soon as it happens even once, you'd think that there would be a big red warning on your dashboard saying "X purchases were refunded because they weren't acknowledged". The fact that the author had to figure it out through user emails and negative growth is strange.

Mobile games get frequent refund requests and devs are more than a few steps removed from the actual accounting from app stores. The analytics might throw an event if there’s a hook for a refund request but Apple and Google own the bank in a black box. Basically devs get a monthly statement with refunds subtracted from their cut.

There's a huge difference between the user explicitely requesting a refund, and Google automatically ordering one because the purchase wasn't acknowledged.

I'm not familiar with the API, and I could see both of them looking the same on the dashboard, but Google themselves should be able to distinguish between the two, and one of the two cases is very likely a bug in the app code. I could not see a case where not acknowledging the purchase would be intended.

This is a great idea. I'll bring it up with the Android Studio team and see what we can do.

I was surprised when I read that he moved back from Fluter to Kotlin, down the line I read it's due to APK size.

I'm happy to hear some developers still care about size esp. for mobile, but on the other end, I'm sad because I'm currently using Flutter for building apps and I'm jealous of that 4.5MB difference :D

I love everything about flutter except the size, and I would have stayed on flutter if it wasn't for the competition's app weighing around 1.6 MB.

Does the size of an app affects something besides the initial download?

Maybe it also lengthens the app loading but everybody here doesn't talk about it because it's well known of android developers, which I'm not?

Yes, bigger apps take up more space in storage.

I feel like most phones nowadays don't really care about a 5mb difference considering they have multiple GB's of space. At least on my phone, the space taken by the actual app binaries is negligible compared to media.

Most people have no concept of how much space things take. When their phone fills up because of too many videos of toddlers bumbling around they assume it's the apps.

One app being 5MB larger is negligible. 50 apps being 5MB larger isn't a huge problem but it is significant. Modern phones have gigabytes of storage but not many gigabytes.

I also didn't see that the difference is just 5MB. Most apps are in the double or even triple digits.

In addition, caring about anything being 3 times larger than necessary stikes me as the right mindset. Individual items may insignificant but they add up. It's certainly better than the electron-ic opposite.

When people fill up their phone, they look at apps, sort by size, and bigger is more likely to get removed.

For users who haven't enabled automatic updates, they're going to delay updates more often for bigger apps too.

There may be runtime impacts too, but that's outside my area of knowledge. I would guess there are better things to look at to improve responsiveness of an app than simply executable size, but it probably doesn't hurt.

You’re always going to be targeting a framework that’s playing catch-up with the native APIs, so it’s not like app size is the only benefit.

Kotlin multiplatform should be a great tool pretty soon to share lots of code between iOS and Android withouth losing on app size. It is mostly business logic you will be able to share this way, but seeing how Compose and Swift UI look extremely similar, the cost of doing the UI twice won't be too bad.

This aside, I am surprised somebody changed framework not once but twice ! That must be a small codebase (or somebody that really loves to experiment).

Shoutout to the people who cared enough to want to pay for it

I kinda thought that myself.

Wait a minute. 5 lines of code were missing in the app, causing users to be refunded. So does this mean a malicious user could decompile the app, remove this code, recompile, and get these auto refunds? This seems a way easier mechanism to get free in-app purchases rather than reverse-engineering the rest of the code.

Great potential exploit. From what I have read, decompiling usually doesn't reproduce the app 100% though so it usually won't run. Is this correct?

How could this be?

Your recompiled version would not have the proper cryptographic signature as you don't have the keystore used by the dev on your pc

The version without those 5 lines will now be forever out there though. Malicious peers can download the apk and get the premium version for free. This feels like a poorly thought out design on Google's part.

Obviously if you decompile the application and modify it you can get the premium features for free. What’s your point?

numerlo is saying that users don't even need to decompile the app - they can continue sharing the buggy APK with each other to get free stuff.

You could also just modify the app to skip the "is premium" check to get all the premium features.

I would hope that there is something more robust for server-handled purchased though.

did the same implementation one month ago (and also missed the acknowledgement step :/) and we are 1 month before release, first thing on Monday morning to add and test it. Thanks dude cheers :D

Seems like this shouldn't be the default. As accepting and then refunding a payment is so uncommon, it should require extra code for that case instead.

I think it’s for the case where the app crashes after accepting payment but before writing the granted asset token to the user’s profile. No real way around that other than two-phase commit.

If you don't deliver the purchased good, a refund seems like a reasonable default. Want to keep the money? Acknowledge you've delivered the good.

The issue is that if you have a bug where the feature isn't activated even if they pay, the user would lose money. It's natural that as you activate the feature you should acknowledge the purchase. Otherwise you could have bugs like double charging and such.

It was an unfortunate mistake by the author, but it makes perfect sense why it happened. The person who made the mistake should lose out.

Little off topic. How was your experience with Flutter? How does it compare with other two?

Building a successful business requires taking responsibility for all the boring details. Complaining that it is someone else's fault wastes precious time and hurts morale.

New payment API, it was something they have mentioned.

A lesson for next time always read blog posts about new APIs you are planning on implementing, especially when it comes to payments.

Just installed your launcher, exactly what I was looking for :)

One change I'd suggest, dragging to reorder custom apps.

Also off-topic, but why move away from React Native?

Alternative title: I missed 5 lines of code and lost $200 (and it's all my fault)

I'd have just added "of sales" or "of my sales" after the dollar amount to make it more clear.

I actually looked at this because I thought Google had refunded money to the author and I wanted to see what had actually inspired an act of customer service from a company known for not having it.

Seems like a PEBCAK

Reply on news.ycombinator.com