Perhaps a stupid question, but why they don’t test the firmware updates internally before releasing them?
This is more common than you think. Only a few days HP update bricked their printers https://arstechnica.com/gadgets/2025/03/firmware-update-bric...
Similar thing happened to Hisense https://old.reddit.com/r/Hisense/comments/18xnmz9/the_latest...
Samsung phones: https://www.androidcentral.com/phones/galaxy-s10-phones-smar...
- The test jig is probably pristine, so no hundreds of hours of telemetry data clogging up the internal storage.
- The test jig might be on ethernet whereas a lot of users would be using wifi.
- The test jig probably targets specific A -> B upgrades rather than testing progressive upgrade across every version that's ever existed.
- The test jig can't cover every permutation of config options.
- The test jig probably only does a bare minimal smoke test after the install, so if the problem takes a bit to kick in, it might not show up.
Not to say that it's certainly any of these, but all are possible contributors. In the coming days it'll become clearer what particular pattern the affected devices follows, and/or clever people with JTAG dongles will reverse engineer the problem and spill the beans.
Like once upon a time, someone established a lab with twenty different units in different states, and put in place a process for validating the releases on it, but that person is long gone, and parts of the lab haven't worked quite right in years, but the parts that do still give a green checkmark, and who wants to stick their neck out and block a release over some baroque process no one even understands, right? It's not like the lab ever seems to really catch a major issue, does it? Just send a :ship: emoji to the slack channel and wait to be assigned your next ticket in the sprint meeting.