"I'll just do this small release on a Friday before heading out" - thought a software engineer before AI-slopification period. Today it was probably some AI agent which fixed all of the performance issues by deleting everything.
I've also used Netdata for years on many servers now. One of the problems with it is its way to fail silently when some healtcheck or alarm is configured badly instead of erroring out on the process start. I understand the reasoning behind it - it tries to monitor by default as much as possible and if some collector fails for whatever reason (let's say you don't have MySQL installed) then it will be just disabled and only way to know that is by looking at the dashboard and not finding it there. I'm only using local agent and not the cloud which means that if server dies totally then there's also no monitoring insights - something to keep in mind.
Creating complex software is not at all complicated, and sometimes it is done right at the beginning of a project, even before solving a single business problem.
Double-checked - no e-mails prior 18th of August from "Google Play Console". Also, double-checked "Google Play Console" (even under "Archived messages") - first message about deprecation I see is from 17th of August (a day before e-mail). Even if there were something a year ago, then 3 month, 1 month followup would make sense instead of 2-3 week reminder.
Maybe it's possible that different roles in "Google Play Console" will receive different type of messages at different time? I'm not an admin for that app and it's possible that someone else has gotten prior warnings, but might have ignored these, because usually they are not with technical background.
But this is also kind of a problem created by Google - why force from 30 to 33 and not force from 30 to 31, then 31 to 32 etc? Gradual API level upgrades would make more sense in my world and probably would be less risky for every party.
What do you mean by that? Are you effectively trying to say that allowing upgrades does not have any risk of attack surface? I'm pretty sure that updating things have also a pretty high risk on introducing new previously non-existing security issues into your code-base/product.
I'm the OP and thank you for thinking along with me here. As stated in numerous replies already then I totally agree that I could have done better in terms of testing things out - of course, there's always room for improvement in that regard. There was a deadline set by Google (again, first time I heard about it was at 18th of August, not before), change seemed trivial at time and since app worked on an old Android version as it was before then I didn't expect it to fail so miserably. Again, I'm not a seasoned Android dev, but have 15+ years of experience in software development in general so I have some expectations how things will work or not and what to expect and be afraid of. I really didn't know that "best practice" is to do a staged roll-out of "99.99999999%" to have a way of partial "yank" possibility of the latest release. To find out that there's no way to cancel/delete a latest release to fall back to previous working version was just something I did not expect in my wildest dreams (I guess this is something you only learn during situations like these). Yes, everyone can blame me for not testing every functionality with every Android version and I do the same, but please open your eyes and understand that the way releases are currently handled by Play Store is not a sane person would do outside of Play Store. Everyone will have a problem like this at one point and I do hope that this article will and thread in here will lower the number of developers experiencing situation similar to this.
I'm the OP and wanted to clarify in case you missed some points - it is a legacy application which does not have any active dev teams on it and needs only developers attention when Google says so and as mentioned by multiple other commenters here the first time I got that e-mail from Google, was at 18th of August. I would not agree that I have been lazy, but instead trying to solve this problem in the time-constraints set by Google and failing to do so because of the inability to put a fix to production and/or pull back current release version. Of course I admit that there's always ways to improve quality assurance.
There has been zero communication towards me from Google until two weeks until deadline. Yes, maybe if I would have logged into Play Console then there might have been some notifications, but there have been no reason to do so until that e-mail (I'm usually not involved with Android projects, otherwise I might have noticed similar warnings via other projects early on).
Actually nowadays it's not that bad anymore. Android browser itself offers installable PWA and there is an event called beforeinstallprompt event (https://developer.mozilla.org/en-US/docs/Web/API/Window/befo...), which can be used to perform PWA installation on user interaction. Of course it's not supported in every browser.
iOS is more difficult since user needs to understand that "saving to home screen" is same as installing "app" and there's no way to trigger it programmatically or help user in any other way than with visual illustrations.
Typical "legacy application situation where no team is assigned to" here. At least I've never seen in my 15+ year career a legacy application, which has a very good maintenance/testing model in place. It will just not work because maintenance/testing needs also resources and updates, but if priorities are in different places then it's not possible. Of course as stated already multiple times - things could have handled in better ways, but this doesn't mean that Google should not allow to stop release at least. Anyway, lessons learned.
It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.
Totally understand that a roll-back is more complex, but it doesn't explain why pulling back/pausing/deleting/yanking a release is not possible either - customers who got the newest faulty release can uninstall & install to get the previous version back (no need to worry about incompatibilities here) and the ones who did not get the updated version yet can stay using the old one until a new fixed version will be released.
A-ha - good tip about roll-out, uninstall & install. Was not aware of that possibility. But yeah, it's still a hack and there should be a better way - just let me delete/cancel latest release even if roll-out is 100% for whatever reasons.
About better testing - there's always room to improve testing, but no way it's going to happen with a legacy application where no active team is assigned. Only these irregular updates mainly forced by Google are done. Unfortunately.
I agree that things could have been done better in many ways. However, as explained, it is a legacy application, which does not see any active development nowadays and it would just not make sense to build such a robust QA. This particular app have been written long time ago by another company and there's not even simple unit tests. That's the hard truth. But still, even as complex setup as you have, there's still going to happen mistakes and the real problem is that there is no way to pull-back/cancel/rollback release.
How would staged roll-out help in this situation for all customers? When end-user gets the faulty version of the app, does he/she have a way of getting the non-faulty version somehow?
Agreed about conflicting message. And even better - e-mail was pretty vague and had to Google a separate link (https://support.google.com/googleplay/android-developer/answ...) for more info. If I could go back in time then I would press that button, which asks for time until November, because as happened to us - it was not possible to go back to previous version in any way.
Yes, that's definitely one side of the problem and I'm not chasing too much backwards-compatibility. My biggest concern in this particular situation is that there is no way (with Android, at least) to pull-back/cancel/rollback release and everything is blocked behind Google's review process. Why isn't it just possible to "yank" problematic release and continue showing previous release as the latest version. That would solve most of the issues within context of this problem.
But how does that help end-users? They would still need to know that they need to use different store etc. It's a pain. At least Android allows (relatively) easy side-loading APK's for the most critical situations, but for iOS that would be a disaster (not sure if Apple's App Store has a way to delete/cancel/pull-back faulty release though).
It was working with Android 12+ before without problems. Upgrading API target level made some flags mandatory (for PendingIntent to be specific), which throw RuntimeException. But as I also wrote in the article itself then this is a legacy application, which does not seen any active development for more than a year and tasks like these need to prioritized instead of other active developments and since app was working after testing then it didn't seem to be that a big of a deal. However, should have been tested on newer Androids too. But even when the problem happened in production then I was not that worried since Google reviews usually take few hours max. The problem is that it has been taken so long for now and there's no way to cancel/pause/delete latest faulty release. That's the scariest part.
Good point, but as I also wrote in the article then one and only reason for doing anything was an API level deprecation e-mail from Google less than 3 weeks from its deadline and it states that review could take a week (never happened before with me though). If I would have noticed the "extend to November" immediately then it would have left more time. But yeah, I'm not saying that this situation could not have handled better, I'm saying that as soon as you make a mistake then there's nothing you can do. And mistakes will happen, even when testing thoroughly with every Android version.
Yeah, that's also true, but at least this kind of problems can be handled by a roll-back or whatnot. Situation we're currently are in does not allow to do anything while we know that bad build is in production and phones are updating to it automatically. That's the worst situation to be in.
TLDR; Written an article about a real-life case-study about Android app deployment/development problem where production version has a critical problem and update has been "in review" for 72h+ and there's nothing else we can do.
If there's some (ex)-Googlers who could help to speed up update approval process then I would be really helpful, if not then let it just be as a warning for anyone else being involved with mobile app development.
I was not even aware that PWA can be submitted also to the store. But yeah, it's crazy that there does not exist a way to yank/cancel latest release so that no more phones would get the faulty update at least. I was not aware of that possibility before it happened and I hope that this article is a lesson to others too if nothing else.
All security headers were supposed to be added by gin-contrib/secure middleware (https://github.com/gin-contrib/secure). However, your comment made me verify that it actually works as it was doing at the time of starting to use that. I noticed that headers were not present - it seems that somehow the order of registering middlewares had made it not work. I've released a new version where they all work https://github.com/jarmo/secrets-web/releases/tag/v1.0.1
Thanks for making me look into it and finding out about this problem.