Hacker News Reader

Comment by Waterluvian | Hacker News Reader

Somewhat related to this ongoing Mozilla plugin saga, are there any infamous stories I should look up that involve huge mistakes leading to unfixable clients? Ie. Imagine bricking your customer's devices with an irreversible buggy update. In Mozilla's case they were able to deploy a hotfix for many, and an update for others. But I imagine there's got to be some great stories about completely bricking countless devices.

View on HN

It's one of my favorite interview questions. I only know the answer because it happened to us once with 5M users on the affected version before it was discovered. It's a very good question to see how people react in a mostly hopeless situation. The only candidate out of 100s to logically get to the solution (or essentially it) was one of the best engineesr I've hired and worked with.

"So you and your team just released a tested and reviewed build, did gradual rollout and numbers looked good - so out to general public it goes. All clients update over the following days. QA discovers, upon testing the next release, a latent bug that prevents the current version from connecting to the update service and running the updater. How do you resolve this?"

I'm super interested to know the answer to this question. I imagine it's more interesting than just uninstall/reinstall.

As said in the other answer, hopefully the interviewee asks how it's unable to connect. If it's an Oculus situation [0] where the signing certificate is expired and you don't have a backdoor like Firefox studies, the only way is a reinstall or a "fix utility" [1].

0: https://news.ycombinator.com/item?id=16541235

1: https://www.theverge.com/2018/3/8/17095414/oculus-rift-softw...

Need more info, why it couldn't connect?

Eg, if it was just a typo in the domain name, just register the domain new name or adjust the DNS record ;-)

Yep, that's how interview questions work. If someone immediately recommends a solution, that's someone who probably needs to grow in their field a little before being given a position where this is a potentially real situation.

And as an interviewer I don't really want an answer. What I really want is to watch your brain grind away at a problem. So ask these questions, even rhetorically. Talk through all. Your angles of attack.

A good signal is when the question is: Who else is on my team?

To play around with a non-technical solution I would consider rushing out a "Version 2.0" of the application that accomplishes the following (if possible within a <24 hour window):

1) Has a functioning update service that fixes the original problem

2) A quick re-skin/theme-change to i.e. a more "modern" version of its original design

3) Whatever performance/security improvements you can cobble together quickly or were in the pipe already

I would do this because I see this as also a business/finance/sales/marketing problem as a technical one. Assuming this bricked the core product, you have to save the business first, then save the quarter, then the week, then the day.

I think this would work because it is much easier to get a user to re-download a version 2.0 with performance/security/design improvements then it is to download almost anything else (imo). Again, this is assuming no saner solution exists. Interested to hear your accepted answer though.

I might not be your target user base, but that's exactly the opposite of what I would like as an user. I am already forced to update because the current version does not work correctly. Forcing UI/design/any major change (especially ones that are "quickly cobbled together" and thus might be buggy) on me in that situation is not nice. I am then left with the choice between a) non working software or b) a major update I might not like.

Perhaps it's good from the company POV because that's how you make sure your users migrate to the new version even if it is still inferior/buggier.

You’re definitely right, my suggestion is business friendly (theoretically) and likely user-hostile depending on how well it’s executed.

I would find any source of displaying a message to the end user.

Most applications have some data that comes from a remote server, whether it's on the home screen, any kind of announcements screen, server-sourced error messages, etc.

I would use that to communicate an update is available, with a handy link.

"I would give my two weeks notice."

So how'd I do in the interview? ;)

I’d love to hear how this was resolved.

My first thought is to email a link to a fix, including a reason and an apology.

Start screaming and rollback?

You make the next version a major upgrade. Similar to Firefox 53?

“My left shoe won’t even reboot”

https://arstechnica.com/gadgets/2019/02/my-left-shoe-wont-ev...

I don’t know if they found a fix in the end

Two immediately spring to mind, Oculus in 2018 [0] and Zune in 2008 [1]

[0] https://www.neowin.net/news/every-oculus-rift-vr-headset-bri...

[1] https://gizmodo.com/30gb-zunes-failing-everywhere-all-at-onc...

Pretty much this with some versions of Samsung's One Connect Mini box - the box (which is the only way to connect HDMI to many Samsung sets) connects to an update service, but during the process triggers a reboot which cuts the update short, which then starts the update again, which then triggers a reboot and... There appears to be no way to get the box back to a working state without replacing it entirely.

I will never buy another Samsung set again. Shipping a TV where HDMI access depends on a proprietary box is unconscionable.

Not unfixable, but there was a fun story about a game that shipped without the ability to be patched, so the developers later "patched" it by buffer overflowing their own EULA: https://www.gamasutra.com/view/feature/194772/dirty_game_dev...

BLU phones released an update that made it impossible to unlock the phones. I factory reset mine rather than muck with downloading another update for the sd card.

https://www.pcmag.com/news/357649/blu-phone-update-is-bricki...

Pretty amazing you could even reset it. If that happened on iOS you’d be out of luck.

Mobile apps broadly fall under this, because update cycles of apps can be very long.

Storytime: App devs built their app with the debug flag on. (Why would anyone use CI anyways?) The debug flag is used for the local developer builds. All API endpoints were pointing to dev/stagings. In order to make this fuckup go unnoticed(otherwise customer retention would be awful + possible media coverage), the dev/staging endpoints were redirected to production, so that they would take the real orders from the broken builds. But we could only do this for our own APIs, not for our payment providers. So we had to ship product without getting paid. :|

Not exactly clients, but the Parity multisig "hack"[0] was pretty wild. All it took was one permissionless call to permanently wipe the shared library of a large bunch of multisig wallets, thereby freezing those wallets forever. Some of the impacted people, including Parity, are still pushing for overriding the wallets to be unlocked, despite no support from anyone else.

0: https://blog.zeppelin.solutions/on-the-parity-wallet-multisi...

Not an update, but expired certificate on Wink hubs:

http://status.winkapp.com/incidents/4c47cdvgl6b4

Imagine if this affected something like FirefoxOS devices and the lack of working certificate was making it unable to fix remotely? I know I'd be annoyed.

Reply on news.ycombinator.com