My product is currently broken and there is nothing I can do about it | Hacker News Reader

My product is currently broken and there is nothing I can do about it

twitter.com

Maybe I am misunderstanding, but if I read this correctly, he says that he pushed out a major v2.0 release with new dependencies just minutes before a launch event that cannot be moved or changed. Even worse, he says that he never ran the prod build, and only in his personal browser.

If this is in fact the case, this sounds like a severe lack of testing or foresight.Testing should include prod builds in multiple browsers and operating systems, and it should most certainly be released at least a few days before any launch events, if you depend on a potentially lengthy review process. He didn't even take into account that the original v2.0 review could have also been delayed.

Maybe I am severely misinterpreting what happened here (apologies if I did), but to me it sounds like a he is blaming Google's review process when he should be looking at his release and test practices.

Releasing to big fanfare the instant the product exists is a mistake and a common one. Deploy it and let some friends try it first.

This is the way.

What strikes me as odd is that the conversation on Twitter seems to be centered around the Google review not being fast enough, instead of the obvious mistakes with the release timeline and testing.

What gets me too is that he doesn't seem to see any fault in his behavior, and only in the review process.

In business and in public, be humble and apologize.

Well he makes me feel better about my salary when I tell my boss to slow down and double check things and use proper processes that I work on.

For me it goes so well that I double guess if we need to be that careful but better safe than sorry I guess.

That doesn't align with the adage of 'ship every six hours'

To quote a wise man with production experience that I worked for (and has retired early).

"It won't crash on our machines. It will crash at the customers."

Problem exists between enduser and enduser's screen

They're holding it wrong?

Where do I put my feet?

This has, for many years, been my favorite gif of all time.

It's the end user using the product: https://media.tenor.com/OC3DCjhCp_gAAAAC/endusers-customers....

Ever heard of Obecalp? Halcyon days, those were

PEBKAC - Problem exists between keyboard and chair.

I'm never saying that in the verbatim lol

> If this is in fact the case, this sounds like a severe lack of testing or foresight.

It also sounds like every single webapp I've been privy to the inner workings of. I can't fault this guy too hard.

Except that this is not a webapp. It's more like an endpoint-deployed application, like an iOS and Android app. Since it's gated by Google, you have to anticipate that things take really really long to deploy or are rejected.

> Maybe I am severely misinterpreting what happened here (apologies if I did), but to me it sounds like a he is blaming Google's review process when he should be looking at his release and test practices.

While you make all good points, you're a career SWE who's already witnessed or made all of these mistakes enough to articulate what he should have done.

His own release and test practices be damned-- if devs can't quickly push a hotfix for a launch bug, I assume there is no framework to respond to anything more pressing (like security issues) in a timely manner either.

If you discover one of your libraries has been compromised, what's the quickest path to remediation in that scenario (that doesn't involve your app being de-listed altogether)?

> if devs can't quickly push a hotfix for a launch bug

This is exactly why he should be very careful before pushing updates. It takes days to put out a fix and deploying on launch date instead of doing a soft launch would terrify any developer.

in my exp you have to fix it yourself if you want turnaround. so either log a bug and open a pr on the lib with a reasonable fix or fork and fix it yourself and publish or selfhost with jfrog/etc

If the first time you run the production build is after launch, and you only do it because a customer told you it doesn’t work, you just learned a valuable lesson the hard way.

Oh the blindness:

> And I sincerely hope that in the future, there will be either a fast-track option for emergencies, a way to rollback to a previous version or really anything that can help prevent a situation like this.

Hmm, I wonder what's under my control to prevent such a situation?

Ooh I know! Building your phone OS and app store, right? Right?

I also have been waiting for a Chrome Store approval the past few days. And also can confirm that updates are usually very fast.

One theory - they're A:B testing/training an LLM-based approval process & taking extra time to write extensive feedback/validations for their AI. Or they're between tools.

I think it's very likely that LLMs will end up doing this review. (They already likely had an algorithm to detect whether anything meaningfully changed etc for version increments.)

Web store submissions don't take one to two hours. I have a popular extension and depending on the changes to your code it can take anywhere from an hour to several days.

edit:

I avoid environment issues like the one in the twitter thread by making use of a single environment (production == development). The backend "endpoint" is configurable within the extension itself and defaults to production. I can test the production build locally, or point it to the development server.

Yup exactly this. Unfortunately I learned many years ago when switching from web development to mobile development - I can never depend on a "rollback" of front end code. Apple has "emergency policies" too, but I can't remember a time it actually came in handy.

Everything we could think of we added a toggle for. If we weren't sure, it got a toggle - just in case.

Also did most of our final testing against "production" builds because that is the exact binary that customers would end up getting.

The problem with a "fast track" option for boosting your priority in the review queue is that obviously everyone will use it all the time, and then it doesn't work.

But maybe it could be a "pay $200 for fast track, limit of one use per two months" or something.

A rollback facility seems like it should be more possible. Not sure why they don't provide that.

Actually if there's a cost then theoretically the whole process can scale up with the fast-track being dedicated, self-funding additional resource.

Purely conceptually if the cost is fully passed on this could scale to any level.

The other solution is to cap fast track capacity or price it higher, so that it's value is roughly aligned to cost. Not super ideal or fair but at least the option is there

While boarding a flight last month, I was in the last group because I refuse to pay for the “privilege” of boarding quicker. I was thinking, if everyone on the flight paid that $37 to board quicker, then who gets to board first?

This whole thing about “premium” stuff feels ridiculous to me.

Kind of like Disney's priority passes that now everybody has to buy to hope to get on a ride at some point during the day.

>Monday afternoon, 3pm. 1 hour to go until launch (4pm local time).

>I receive an email from a customer that mentions Tailscan has not been working for him after the v2.0.0 update that was pushed out just minutes before.

Pushing an update an hour before launch was... unwise.

https://nitter.net/Erwin_AI/status/1724639265592906210

So a completely broken product went through the review without problems. This makes me wonder: what do they actually review? Do they even try to run the thing before approving?

They’re not QA testers they’re making sure you follow the store guidelines.

I simply do not care about the plight of developers who get caught up in Google or Apple store policies. Stop tending other people's gardens.

Whats the best direction for someone who wants to write apps for either given your comment?

Accept that the only reason you want to write apps for either is to get customers, i.e. approval or money. Become enlightened. Write an app for the craftsmanship, ignore users until one day they magically appear.

What if I want to write apps for myself that solve random niche issues I run into that would be worth it just for me to learn to resolve and pump out a solution for that so, at the very least, I (capital i) solve one of my own problems (craftmanship?)

I feel like we're not saying terribly different things here. That is one of (if not the greatest) motivator. I hate the idea any app fev can shittify their product and take away a foundation you rely on cuz #reasons.

> What if I want to write apps for myself that solve random niche issues I run into that would be worth it just for me to learn to resolve and pump out a solution for that so, at the very least, I (capital i) solve one of my own problems (craftmanship?)

Write it for literally any other platform.

Developing anything inside a walled garden always comes with the risk that you might later end up on the wrong side of that wall.

Ya but with sideloading that's presumably not such a sure thing anymore, no?

Like why can't I write for AppStore normally and then if/when there's a different context, be able to write for AltStore or whatever the broadened appstore market is like? I don't know what kind of alternative platform you're actually proposing and I'm not knowledgeable enough to actually be able to engage you on this beyond the basic concepts I've run through over this thread

I also came to the realization that apps dont have any magic functionality you can't emulate or copy wholesale, just that they might package it up in a way that makes sense to either buy in perpetuity or subscribe where necessary and buyable solutions don't currently apply or exist

Bit harsh here. It's a css tool, not a core app for a large bank. Erwin gets points from me for being in the arena trying things. Anyway, am I correct in assuming an acceptation server would have prevented his problem?

Reply on news.ycombinator.com