Hacker News Reader

Comment by tomstokes | Hacker News Reader

Two important features I insist on for products I develop:

1. Staged rollout of firmware updates. It’s common practice for apps and software but for some reason it’s less common with firmware. Rolling out to 1% (or less, depending on scale) of devices and waiting a day is cheap insurance. Side note: Build a good relationship with customer service people so you hear about these things immediately.

2. A failsafe firmware reset back to factory state. Some sequence that resets the device completely back to the way it was when it came out of the box, firmware included, as a last resort. In conjunction, your automated tests need to confirm that every factory firmware you’ve ever released can update to the latest firmware.

View on HN

> A failsafe firmware reset back to factory state.

This doesn't work if your threat model includes denying rollbacks to prevent exploiting bugs in old firmware. I'd love to be able to roll-back firmware on some of my devices to allow me to "jailbreak" them using old firmware.

In some cases your newer firmware may be blowing e-fuses that prevent old firmware from functioning. See the Nintendo Switch, for an example.

To be clear: I think this is anti-consumer and wrong, but manufacturers absolutely do it.

Edit: I also think it should be illegal, by way of consumer regulation. I don't think consumers should have option to waive their right to manufacturers not damaging hardware they own.

This doesn't get enough attention, waaaay too many of these issues are traced back to the vendor trying to "prevent" someone from using their product in a way that they don't like.

Why else would a soundbar need updates anyway? It either performs its well defined functions when you bought it or they sold you a device that doesn’t input/output sound.

Updates for these types of things always fall into three categories. Either they’re gimping some unanticipated usage, they’re trying to insert ads, or they’re trying to gather more usage data.

Maybe a new codec? New streaming app support? New wireless protocol? CEC bugfix?

Yes, all of those are in the realm of possibilities, but has it ever been the norm?

In my experience, products like this are only get updates when the company finds a way to extract more money:

- add more ads

- add more ads that pretend not to be ads

- to remove functionality, so it won't cannibalize sales of more expensive product

I actually picked up a Samsung soundbar for my mom this past Christmas and there were quite a few negative reviews. Usually around the soundbar dropping its connection. However diving deeper on them seems to revealed that the issue was resolved with an update. It's not super smart though and needs a USB drive or phone app to update. So it has prevented this situation from happening.

Considering the soundbar connects to a TV, console, phone, etc that are constantly releasing new versions and upgrades it makes sense to build in the function to something as simple as a soundbar to fix bugs and compatibility issues.

Samsung doesn't have the greatest track record with updates though so obviously you don't want to jump the gun on these. Hopefully not a Galaxy Watch 4 situation where they need to be mailed to Samsung to be reset because they didn't think about this during the design phase.

More hardware is sold at cost or at a loss, compensated with ads. I don't like the model either, but that's how it is.

If price isn't the only factor for some, it is for many who would otherwise not buy these things. Sellers picked up on that long ago.

Other comments wish to see regulations, they can't outwit those marketing tricksters. For profit enterprise can, and will offer more alternatives with bigger stamps about privacy, ad-less certified and whatnot.

It’s the norm because people rather buy one single product that does it all.

The alternative to an all-in-one sound bar is having regular 5.1 speakers, a nice receiver, a nice streaming box, and maybe a dumber TV and you will have absolutely the best setup but it’s a lot of putting pieces together, more space usage, and either money (if you want it right away) or a lot of waiting (if you want to get it used).

Even dedicated receivers have software updates now. My Onkyo receiver had an update that added Dolby Atmos support, for example.

I had a Yamaha that had a dtsx firmware addition upgrade after it shipped. Not sure if it wasnt ready at product ship, or some way to avoid licensing fees, but I dont know how they would track who upgraded as it wasnt network enabled.

Sennheiser Max has a full computer and os running inside, they can upgrade it quite a bit. Biggest limitation on the device is HDMI 2.0 preventing 20gbps video passthrough of hdmi 2.1, however they should be able to add new audio codecs.

> Why else would a soundbar need updates anyway? It either performs its well defined functions when you bought it or they sold you a device that doesn’t input/output sound.

Unfortunately there are soooo f..ing many devices out there that don't follow the specs, no wonder given how long and complex alone the Bluetooth specifications are, and HDMI/HDCP (which a soundbar with ARC support needs...) is even worse, and don't even try to get me started on CEC because that is an even bigger pile of dung, or stuff like GPUs that run HDMI over DVI, MHL or USB-C in DP mode and god knows what else people expect to "magically work" with a 5 dollar adapter they got off of Alibaba. And no, "audit products to follow the specs" isn't a foolproof solution either. That means that everyone has to deal with everyone else's quirks and at least the most popular devices and their manufacturers have to supply firmware updates to react upon reports of quirks.

While I agree with what you wrote

> [...] GPUs that run HDMI over DVI [...]

I thought HDMI and DVI use the same signalling (at least the 'digital part' of DVI, was it DVI-D?), just over a different connector?

In my memory only the connectors competed for adoption, and Home Entertainment industry opted for HDMI and the PC-industry opted for DVI, while the signalling was not contested (besides DVI also being able to carry analog signalling with full spin-out, and HDMI carrying audio instead). My memory might not serve me well here though.

I never thought HDMI would win :( but it makes sense I guess - Computers/their use changed :(

Even without the relative size difference of the TV and PC industries, the HDMI connector is simply more compact than the DVI connector.

Now Display Port vs HDMI is a more interesting competition and it would have been nice to have a clear DP victory here.

Sibling mentioned CEC fixes— this one is huge. CEC is lovely in concept but I ended up having to disable it completely across my setup as there was just way too many bits of weird behaviour with devices turning themselves on and then switching the TV or AVR to their input apropos of nothing.

I feel like CEC tried way too hard to be magical instead of exposing enough control for the user to be able to block certain commands from problematic devices, or even just designate that device X will always be the boss in a particular setup.

Absolutely this.

The frustration when I turn on the Steam Deck and the Apple TV goes

"Look at me. Look at me! I'm the output now"

Yup, game consoles are ground zero for this. I hit the button on the PS5 controller only to have the receiver and TV power on, then the PS4 wakes up for some reason and then switches the AVR to its input.

My Sony UHD player also seems to want to grab the input sometimes too, so maybe it's Sony that's the source of the problems haha.

And again, it's all just so maddening because it feels like it would go away if I could be like "Hey, AVR should never send power-on messages to its input devices." Because then I would just power on the device I actually want to use, it would turn on the AVR and TV, and we'd be golden.

I turn off CEC all the time and my tv refuses to acknowledge it if I ever unhook the device or HDMI. Always defaults back. Drives me crazy.

Highly recommend https://www.amazon.com/Lindy-HDMI-Adapter-Female-41232/dp/B0... -- I have a couple and it's solved this problem for me completely. I hate how unpredictable CEC is when things go wrong, on top of the ridiculous 3 device limit.

Even better: I have some sort of Useless Machine[1] bug where turning on the TV will power up the PS5, which then puts itself to back to sleep.

[1]: https://en.wikipedia.org/wiki/Useless_machine

Oh I've definitely had this one too, where the TV powers up to the "I'm going to sleep now lol" screen from the PlayStation.

> And again, it's all just so maddening because it feels like it would go away if I could be like "Hey, AVR should never send power-on messages to its input devices."

Yeah, that sounds a weird "feature" in the first place.

If I manually turn on the UHD player/Chromecast/PS5/whatever, it makes sense that the TV also turns on and switches to the respective input.

I could also sort of imagine that if I switched the TV to some input source, it might be convenient if the device connected to that input turns on. (Not by a lot, though. You need the device's remote/gamepad/whatever anyway to tell it what to do, so the one button press saved doesn't really buy you much.)

But what makes no sense for me is the TV turning on all input devices when it's being turned on itself. When would you ever want to have the PS4, the PS5 and the HD player running, let alone as the default behavior?

That sounds like a genuine bug in the TV.

(Also, you sound as if you have some sort of "2 <-> n" setup with n input and 2 output devices. I have no idea how CEC would even be supposed to behave in such a setup. Would an input device turn on both output devices?

It's a conventional setup:

TV <- AVR <- PS4, PS5, Switch, UHD

I suspect the issue is largely with the receiver (a VSX-935), as that's seemingly the component sending a turn-on signal to its inputs.

If I could, I would have probably run everything to the TV and just done all the audio over eARC, but the TV is on the other end of a 50' HDMI cable, so I definitely need the receiver as an in-rack multiplexer.

Ah, that makes sense.

I have a laptop, steamdeck, Nintendo Switch and chromecast all connected to an LG TV and all the ouput switching and remote pass-through works as expected. Maybe just a lucky combination ?

  > Why else would a soundbar need updates anyway?

No matter the device, software rots.

Not because the device changes, not because the software changes, but because the world does

Also, time-to-market pressures can result in initial shipments having (minor but not showstopping) firmware bugs. Post-sale firmware upgrades can be beneficial for the customer.

And the obvious solution is to isolate the device from the world. Most of my stereo is isolated from “the world”, and some parts are close to 30 years old. Why does a soundbar need contact with the internet?

Modern soundbar are bugged Bluetooth enabled, also with ship with interfacing protocols, while legacy bluetooth/wifi drivers are ok, protocols just break

Innocuous product features like streaming music, integration with Alexa/Google, connecting to TV and other speakers via wifi. Oh and collecting analytics data and selling to ad networks...

Just because you want to keep using old tech doesn't mean everyone else wants to.

I prefer to choose myself when I buy new stuff, not let the corporations decide that for me when they decide to brick my old stuff.

That kinda defeats the point of having a device. Sure it works in some cases but we're talking about a soundbar here and that has to interact with other devices. It's whole purpose is to interact with other devices.

Even if it doesn't need to contact the internet you're still going to want it to connect through cables. There's good reason to connect through bluetooth.

But why should it contact over the internet? Well it sure is nice to be able to stream music from my NAS. There's utility in that. There's also utility in the parent company updating firmware to support new audio codecs. Or to support new algorithms. If my device is gaining more utility, that's a great thing! And of course, if it is connected wirelessly in any way (including bluetooth) I sure as hell would like updates with respect to security.

Without this, the thing becomes e-waste. The environment moves. Time marches on. No thing can exist in isolation, no matter how hard you try. Again, software rots, not because the software changes, but because the world does.

But that's not the problem here. The problem is abuse of that power. It isn't for the benefit of the customer. The problem is managers pushing to release before things are ready. The need for speed with no direction. To not even consider in the calculus of decision making the tremendous costs of when things go wrong. And how this lesson is never learned despite facing the problem time and time again. Issues like this now cost tons of engineering hours, tons of lawyer hours, and ultimately will cost tons in rebates and refunds. How many weeks of work is that equivalent to? Sure, it doesn't always result in catastrophic failure like this, sometimes it results in smaller failures, sometimes small enough they can be brushed off. But those are still costs that no one considers. That's the problem here.

In my case, my stereo is connected to an inexpensive Airplay adapter.

So I do get all the advantages of a connected device, but if the adapter is bricked, I can easily replace just that small device. And more likely, when there’s a new standard, most of my equipment is unaffected.

s/soundbar/airplay adapter/g

I believe you're missing the forest for the trees. My argument is invariant to the specific device we're talking about.

No, you are missing my point. In the same way as we do (or at least should do) when we develop software, we isolate the volatile parts from the stable ones. The loudspeakers have looked the same for decades. No revolutionary changes in amplifiers in a long time. The same with DACs. That means that when a software update bricks my adapter, or a new much better standard comes along, or I decide to leave the Apple ecosystem, I only need to replace one small part of my stereo system, not all of it.

This should be done internally to the device. I do agree that nothing you do should affect how speaker sure input is processed. But if you want those other features it's much more convenient to integrate them on device or rather place them within the housing as there's lots of empty space.

With electronics you can still isolate functionality like in software how we wrap things into functions. But like software sometimes we need to break that for optimization. Think like Apple M chips. They do it in the most annoying way, but integration is helpful. Ideally in a speaker though you should be able to fuck everything up and still allow for raw input.

As for the Apple thing, well that's a bigger issue because we really should be using open protocols and fuck walled gardens. Walled gardens are part of the problem we're talking about

Why does a soundbar need software? An active speaker with a jack plug would work just fine

Welcome to the world of embedded devices.

Sure, you could do everything through a static circuit and require things being fed with speaker wire. But if you add a microcontroller you're going to be able to do much more, get better sound quality, and protect your equipment. Do your speakers have batteries? Do they plug into wall? Either way you can better control power levels. Do you want to boost bass? Fix corrupted signals? Do you want to process signals from anything other than a bare wire?

Sure, you don't need a microcontroller in a speaker. But we also don't need them in our cars. You don't need them in your fucking kettle. But personally, I find them useful and considering how cheap they are it's worth the basically $0 increased price.

See my other argument. The issue isn't that there's a microcontroller in the speaker. The issue is bricking the device. Don't confuse the means in which a bad actor operates with the bad actor themselves. You'll never stop the bad actor by just banning everything tool they abuse. You'll end up with nothing.

>get better sound quality

That just isn't true though, is it? How would a microcontroller add sound quality?

Imagine your signal comes in degraded. Some extra noise on the wire because it is passing next to a faulty wire in your walls or something. You can then do a FFT (example) and pull out the noise and rebalance the signal. Maybe an easy way to think of this is with radio since you're very used to dealing with static in that domain but fundamentally there's nothing different than signal coming through a wire other than the technicalities of the medium through which it's transmitted.

There's much more signal processing you can do besides FFT btw and many can improve signal quality and thus sound quality. Even something like a built in equalizer. Sure, you can do this all with hardware by creating all the right filters but you can do more in a smaller package with a computer

At least in theory these Samsung sound bars are supposed to adapt to the listening environment to more accurately render the intended surround sound. They also have various non-trivial inputs (including wireless ones) as well as support for additional real speakers and subwoofers which again might need changes for compatibility.

Of course they could be designed to be simpler and have whatever input device is used (e.g. the TV) handle fancy features like mobile phone support.

> Why else would a soundbar need updates anyway?

Because for free you only get the first 15 levels of volume. If you want to get to 25, you need to pay a subscription.

I thought it was obvious... how does the seat heating work in your car? /s

We've solved long ago mass manufacturing challenges. Today's problem is to sell.

Upvoted, but I'd pay a subscription to restrict a neighbor to the first 15 levels of volume out of 25 sometimes :)

While I agree with your broad statement, I have a TCL (with built-in Roku) TV that has a bug in the sound processing. Either it becomes very quiet, drops out completely, or comes in and out with a lot of stuttering. Happens irregularly, typically though not always weeks apart (though on no schedule I've identified), solved with a reboot of the TV (which of course can't just be done by turning it off and back on - you have to select "restart system" from the menus).

I owned it for at least six months before this occurred the first time.

In theory, I could do a USB update of the firmware and hope that fixes it. In practice, they want my serial number to let me download it. No thanks, I'll pass, even though it's never been connected to WiFi or Ethernet and never will be. I'll just reset it every once in a while.

> they want my serial number to let me download it.

Out of curiosity, why is that a problem to you? Granted, it is strange; I went through the process for my TCL Roku who's wifi stopped working (still not fixed, and now a second, 3yo TCL Roku has bricked itself. nice!)

I don't care in principle, but it's not just that. You have to give your serial, you have to boot the TV to the update, which then sends a challenge-response to their servers that must be correctly answered (you use your computer for this, so the TV isn't actually on the internet) for the upgrade to proceed.

I don't know what's in that data. And if I don't know what's in it, I'm not inclined to proceed; you might need my serial number to know if you're giving me the right software, but you don't need challenge/response for that. They sold me a cheap TV in hopes of collecting info on everything I watch, whether via Roku or just screen analysis. No thanks, and I have no interest in making it easier for them to break into my WiFi. I'm sure it would connect itself automatically to an open WiFi.

It's a little paranoid, but they really are out to get us (or at least our data).

To install an AI update you didn't ask for, do not need and cannot turn off?

A lot of consumer products ship with half-baked software and/or firmware. I wish Polk would fix the bug(s) that cause my soundbar to freeze and need a reboot several times per week. But it's an old product that's not longer sold, so I'm probably SOL.

The problem usually aren't vendors. The problem usually are rightsholders - the movie/TV series industry still didn't get the Spotify memo, and the console game industry... well it's hard to say they don't have a point insisting on serious DRM given how rampant piracy becomes once there's an easy-enough root method available.

This is an undersold part of the story

It's not only media companies with DRM

IoT integrations like Alexa come with numerous security requirements that are often good ideas in theory but lead to hacky workarounds to meet certification requirements

The massive success of Steam points otherwise.

In what way? Console makers wouldn't gain anything by weakening DRM and making devices rootable. It's not like they are making that much money from device sales.

Of course then you have MS which basically just turned XBox into a cheap but totally locked down gaming PC (since there are very few Xbox exclusives these days).

Steam is a very convenient and beloved marketplace but that doesn't mean it doesn't have a solid DRM and anti-cheat measures built in.

Steam's DRM is a joke. Removing it is as simple as replacing a library.

It also doesn't cause (intentional) incompatibility problems like HDMI DRM does.

Is this the Spotify that is a broadly unprofitable business, which is why it's so desperate to enter into new ones, or the Spotify that has DRM?

Spotify made 1 billion $ of profit in 2024. Hard to call that unprofitable.

My point is, it (and Youtube) killed piracy for the most part when it comes to music. Trading CDs full of mp3s used to be a sport in school a decade or two ago, these days why would anyone even want to invest the time when Spotify has everything anyway at a price point school kids can afford it?

Netflix used to become the same thing for movies, but the greed of studios killed it and now it's more expensive to have the large stream services than cable TV.

> the movie/TV series industry still didn't get the Spotify memo

I'm not sure that's really a memo I'd like them to get. We don't need more subscription services where you don't get to own you content and everything can be taken away at any time.

Exactly. If your company's threat model considers its own customers as attackers, you're the baddies.

Not always. There's a time and a place for including end users in your threat model. These would include scholastic and carceral settings, where in both cases the end user may, as an example, desire access to resources that have been deemed inappropriate.

> scholastic and carceral

Same thing.

> deemed inappropriate

Ooh! Deeming! Can I deem too? Huh? Can I? I have a number of candidates.

I disagree that a software in a school setting should see students as adversaries. Cheating is a much higher level problem that is better dealt with education and negative reinforcement. After all, those students will need to become participants in a society where we definitely don't want this level of mutual distrust around every corner.

But in any case, students are usually NOT the customer here even if they are the end user.

This is a good reason for manufacturers not to deny rollbacks, and a good reason not to have e-fuses.

Blow the fuse after its confirmed working. Or always allow a one version rollback.

Im not a fan of firmware lockdowns but I understand other people may value security over moddability.

At very least, it should be two partitions: previous firmware and current firmware.

even with that "requirement" add special minimal recovery that can be booted with special buttons sequence by bootloader and allows some form of flashing signed firmware.

this should be especially trivial when your device have some usb ports.

you can keep all requirements of only newer or the same version of firmware to flash, with all refuse checks.

if you mess up, you can allow consumers to flash fix using regular pendrive

Yes, they do it, but usually in devices where it's basically part of DRM. I don't think engineers put that much though in security of soundbars.

Yup! Depends on what's a higher priority: Preventing catastrophic destruction of the device, OR, "protecting" some IP from ultra-small-scale piracy, even though ultimately anyone bent on piracy will be able to pirate anyway.

Clearly the latter is heavily preferred by most companies.

Big part of the UBNT vs Cambium dispute. IIRC UBNT won in court, but just to prevent the Cambium firmware being installed on their hardware the next few firmware versions fixed it so that it cant be easily reverted.

Whats worse is that a lot of the affected hardware was near or EOL anyway, so Cambium was simply helping rescue devices headed for the scrap heap.

Sometimes they do it because it’s contractually required if they want to get access to proprietary standards, for example to allow them to play copy-protected content.

Copyright and patent have morphed into evils that drive anti-consumer and anti-competitive behavior, and have driven a “subscription” model that allows rent seekers to achieve their wildest dreams.

Blowing efuses is a destructive action and it should not be legal for a company to destroy parts of your electronic device that you paid for

I think the correct way to do this is to allow a rollback to the immediately previous working version. Before updating, write current firmware to failsafe data storage, then do the update. Then a firmware reset sends you back to the last good version. I'm pretty sure this is already done by many hardware and software manufacturers, such as me.

Yes it does work… with an A/B update system.

Android systems can do this today. After an orderly shutdown of new software, then it can mark the new stuff as good and not allow older software to boot.

The funny part is the Samsung update that bricked a10 phones was a update to smart things, so it couldn't use the Android A/B capability to roll back lol

But then at least have backup firmware of the one you want to update, so you can go one step back in case of errors.

Is that applicable here? We're talking about speakers. For most/low security devices, a firmware rollback, or a firmware-download mode, are fine. In this case, it would probably have prevented millions in losses, with the risk being a...jailbroken speaker?

This practice should simply be illegal or at least make the manufacturer liable for a full refund plus interest. We shouldn't let manufacturers brick devices that we own.

Most companies don't do this because it's not one of their organizational priorities to have reliable updates. The infrastructure is usually custom built and maintained by a couple of folks who have a dozen other responsibilities they're told are more important. Testing is usually limited by hardware availability and release velocity. "One of every board revision we've ever produced" simply isn't available and waiting two days to run through every firmware version before you release updates is a conversational non-starter with the PMs.

There are commercial offerings (like mender.io, never used) that basically specialize in providing rock solid update infrastructure, but that again takes investment and organizational priority that doesn't exist for non-feature code.

Different industry, but I (a long time ago) worked in a place that built scientific instruments.

> "One of every board revision we've ever produced"

The, ah, "special" people we had running engineering didn't even put in the work to be capable of the software querying the board rev. We had to play games like running certain motors past a position limit and seeing if there were limit switches there (or not) to guesstimate board revs.

I'm guessing stories like this are common.

I'm working on embedded systems and I've seen and heard some horror stories just on the device's side. Piles and piles of pre- and post-reboot shell scripts filled with race conditions against the system's services and themselves. When these break, if you're lucky a factory reset is enough to fix the system, if you're unlucky they become field bricks.

I'm trying to buck the trend though and on the new embedded system I'm working on, I've specifically designed the upgrade system to be as reliable as I can make it. It goes something like this:

- The new firmware is downloaded to the secondary application slot.

- Just prior to rebooting, the entire state data of the system is serialized as a document and stored on a flash partition.

- The upgrade flag is set, the system reboots and MCUboot does its thing.

- The new firmware finds out a upgrade happened, clears out all the data partitions, restores from the document and then clears out its partition.

The system is basically sanitized and restored after each upgrade. It's also the same codepath that handles saving and restoring the system's configuration by the end-user as well as settings management. If the document schema is for an older version, run the N-to-N+1 schema upgraders on it prior to applying instead of trying to patch the system in-place. If something goes horribly wrong, flip a jumper to trigger the heavy-duty sanitization that nukes the entire external flash (internal flash only contains the bootloader, primary application slot and factory parameters so it's essentially read-only once the application boots).

It might be hubris, but I hope it's good enough that I'll never see a bricked card that can't be resurrected by a factory reset with this project (assuming no hardware damage, no internal flash corruption and no bricking firmware getting signed with production keys seeping through the cracks despite all the checks in place).

add a watchdog timer to reboot automatically on failed upgrade as well.

We already have a watchdog timer. We could automatically trigger a factory reset after N bootloops following an upgrade, but it's up to the end-user to decide to flip the switch so we won't go there.

I kept the summary short and simple, partly because that product isn't out yet and also because I don't want to bury the lead with a lot of extraneous details that we do take into consideration, but are irrelevant to the big picture idea of an upgrade method that factory resets the card and restores its state with a codepath shared with the end-user save/reset and configuration mechanisms.

That's a strong start, but be careful if your system ever evolves beyond a single logical processor. You'll need additional orchestration to have reliable updates in a distributed system with semi-independent processors. The update on one might succeed, while another fails. Depending on when the old images were produced, the new images might not be able to talk to each other. Depending on their relative roles in the system (e.g. one sets up the power supply or network for the other, or acts as the time master to do certificate validation) this may or may not be an easily fixable issue even if each system locally thinks it's okay.

This sort of functional interdependency has become increasingly common in embedded these days with heterogenous SoCs.

One thing I've seen before is to separate downloading from rebooting, broadcast the manifest for the updates between all the independent processors (all updates need a declarative manifest for so, so many reasons) to check locally, and only proceed when they all agree. Rollbacks are initiated if they can't see everyone with their expected versions afterwards.

Still isn't perfect either.

Fortunately, it's a single no-frills MCU running the Zephyr RTOS. It does communicate with another system, but they are so very loosely coupled to the point that we really don't care whatever is running on the other side.

I won't get into details, but in some of the horrors stories I've heard the distributed system happened to be entirely software in nature. There are plenty of creative ways to mess up an upgrade on a uniprocessor system.

I completely agree with both points and would add a third: design for offline use first (maybe treat every OTA update as - this might be the final version this device ever receives). Products should work perfectly fine without an internet connection, heck that's how they worked until 5-7 years ago. Core features should never depend on cloud services, and updates should be opt-in, not forced.

Offline first approach respects user autonomy and creates a natural safety net against bad updates. Plus, it means your product keeps working even when servers change or get shut down years later or a nuclear war happens. Sure, connectivity has benefits, but a speaker's main job is playing sound, not phoning home. Building offline-first also forces better engineering decisions about longevity and graceful degradation.

It's so hard to find any offline-first apps/devices nowawdays, which is sad to see in a world of algorithms and AI.

This whole situation reminds me of this: https://programmerhumor.io/linux-memes/thats-the-attitude-sa...

But you see, the problem with offline use is the manufacturer can't claw back value in the future. How will you keep shareholders happy if you can't arbitrarily push ads, hobble existing functionality, or impose a new subscription service?

Exactly - that's the flaw in trying to extract infinite growth from finite products. We've turned durable goods into rental services without consent, all to please quarterly earnings reports.

The tragedy is that "respecting customer ownership" is now seen as leaving money on the table rather than building lasting brand loyalty through quality.

I get the sense that #2 is viewed as a risk for DRM, given all the work that goes into preventing firmware downgrades to potentially insecure firmware. Specifically thinking of the Nintendo Switch[1] that goes so far as to blow fuses on each firmware upgrade!

https://news.ycombinator.com/item?id=23534793

eFuses were already on the Xbox 360/PS3 generation. Smartphones also use them to lock out proprietary photography algorithms if you unlock the bootloader.

https://en.wikipedia.org/wiki/EFuse

Sonos completely missed the boat on these two simple concepts as well.

See their new app debacle which coupled a non-reversible firmware update that made the hardware incompatible with the old app.

Great points! As an addendum to this, if #2 becomes untenable for whatever reason (such as a vulnerability in the factory firmware image), then this #3 would be good to strive for as well:

3. have a set of conditions to mark the running firmware image as "safe" and have it become the new fallback firmware image for this scenario. That way you can have a recently up-to-date firmware version constantly trailing the new ones

IMO this is a terrible idea for many reasons but the most important of which is: As a consumer I should have the right to have my device revert any b.s. update and get my setup to how it was the day I bought it.

So many companies have begun rolling out updates that makes the device I purchased call home before allowing any user functions and if/when that server goes down my device becomes a brick. This behavior essentially invalidates my ownership of the product and renders it to a service, provided at will by the manufacturer.

Your idea ensures my device will one day become a brick as soon as the manufacturer decides to mark their update requiring internet check-ins “safe”.

If you think I’m exaggerating check out Louis Rossmann‘s YouTube channel.

FWIW, my background is in B2B hardware and that's the perspective I am coming here with. Out of curiosity though, how do you weigh your value of control vs. security vulnerabilities? Modern speaker systems allow some form of wireless connectivity, so there is bound to be something and not all consumers will be savvy enough to keep up with security updates on their own.

My thoughts on security vulnerabilities is that they exist on any out of date firmware and that should be expected. I’ve never rolled back to factory settings and assumed that this device is now exposable on a DMZ.

Specifically I’m talking about consumer devices, which are almost always behind a NAT config + firewall. If your soundbar has a vulnerability it’s pretty much irrelevant if someone has already breached your network.

If we’re talking about enterprise networking equipment, I still stand by my concerns that the the owner should be able to revert back to stock but the burden of responsibility is on the technician configuring this device, not the manufacturer.

It seems to me the mentality has become that since end users tend to be bad at system administration, they shouldn't be allowed to do it, for their own good.

I reject this mentality. I don't think it's necessary or desirable to make it impossible for people to do things that have negative consequences for themselves. Put a "here there be dragons" warning on the firmware rollback, bootloader unlock, or similar dangerous operation and let people take responsibility for the outcome.

In the case of consumer devices, most people won't even try those things; those who do risk further problems for the chance of a better outcome. In the case of enterprise networking equipment, there's an IT department that, in theory has the skills and resources necessary to make good decisions about technology.

There will always be security issues, so "but security" is not a reason to prevent a consumer from doing whatever they want with a thing that they purchased from you (I'm of course just speaking morally/ethically here since there's no legal provisions preventing that in most places).

If I pay you for a product, you have no moral right to tell me what I can and cannot do with that product, up to and including messing with the firmware, installing known-bad firmwares, wiping it and building my own firmware, whatever I want. It's mine, I paid for it, stop violating my private property rights.

I think I agree with you generalle but just from a logics perspective, this is a bad argument:

> There will always be security issues, so "but security" is not a reason to prevent a consumer from doing whatever they want with a thing that they purchased from you

Just because there will always be security issues doesn't mean you shouldn't try to take care of the low hanging fruit.

Not the person you replied to, but I'm literally pulling wire again to avoid dealing with that dichotomy. And hardware developers that think OTW firmware updates are a neat idea >:(

Unfortunate you'd need to weave that all the way through the whole product stack in order not to end up in a state that looks like it's working at first glance but actually isn't doing what it is supposed to - like everything running but not showing an image, or everything running except networking is dead (-> also no further updates possible), or (remote) input devices, etc etc

From the manufacturer's point of view, a sufficient "safe" state is "can receive and apply a firmware update" -- worst case scenario you can always push out a new re-signed and renumbered version of the older working version.

Network connectivity would need to be in the set of checks to determine if an update was successful. Also, there should hopefully be QA. If you only have one smoke-test for a firmware image it should be whether or not it can upgrade/downgrade a new image from that one.

This is what everybody wants, but almost nobody does. Time to market, etc.

You need to have the firmware equivalent of a platform team.

It's common now for medium and large companies to have some variant of a cloud platform team: People responsible for shared practices, infrastructure, and processes in the cloud.

Smart hardware companies have done the same for decades. You have a firmware platform team that handles things like update protocols, recovery protocols, testing checklists, on-device OTA update architecture, and other critical functions.

When you're a company like Samsung that continuously releases and develops products this actually increases your time to market rather than decreasing it. You let each product team focus on the parts of the firmware that make their product valuable and free them from having to roll their own update systems

Samsung has multiple such teams. In my experience with the broader industry, platform teams are usually less than a dozen people who own millions of lines of mostly-external code. You don't usually get the luxury of careful deliberation and comprehensive testing because you're doing too busy putting out fires and chasing down manufacturer errata.

Samsung might be one of the good ones, but sadly most hardware manufacturers treat firmware and software like just another line item on the BOM. Like a screw or a silicon gasket: Source it from some "supplier," spoon it into the product somewhere on the assembly line, and then never touch it again. I've seen a hardware manufacturer that doesn't even use source control or branching. When they have a new hardware product, they take the software that is closest in functionality, hack it until it works with the new hardware, and then set the software back on the shelf until next time.

It's almost exact same thing as purchasing an insurance.

If the management folks have personal health insurance, surely they must understand the concept and the need. And this is a much better deal because unlike actual insurance this is more like "invest once, enjoy forever" type of thing. And multi-stage boot chain, recovery partition and staged rollouts are not some rocket science that needs some serious expertise.

Yet, here we go. Humans are not really rational actors after all, and collective humans are even less so.

I suppose the closest equivalent would be motherboards with dual BIOS.

There if something goes wrong during an update, you always have a backup BIOS with the previous version (not necessarily factory settings). If the system fails to boot, it automatically switches to the backup BIOS and restores the main BIOS to the last working version.

For this $1500 street price soundbar, I'm wondering whether they consciously decided not to invest in BOM cost or software effort that would help avoid bricking.

I'm not sure I understand various industries' conventions...

While interviewing for a principal engineer job, I was meeting individually with a bunch of team leads and managers, and one engineer asked how would I design firmware updating for the company's product (which was more critical, complex, and expensive than a soundbar).

I assumed they were probably trying to see whether I would throw in some robustness/resilience (not oversimplify it). So I sketched it out, while hitting notes like diffs, downloading and assembling in staging space, imperfect networking, having at least two firmware "slots", backing out upon boot loop or failure soon after boot, gradual deployment to installed base, contrasting with some less-critical consumer product firmware update practices, etc.

(Either that was a bad answer, or they got distracted thinking about something I'd said, because I was getting odd subconscious backchannel cues, and they were unresponsive when I tried elicit more requirements or guidance about what they were looking for. Maybe there was some standard embedded systems programmer canned answer that I was supposed to recite (analogous to the Web brogrammer 'system design' interview), and they couldn't think of how to nudge me towards the shibboleth without saying it?)

#2 has been a godsend in the custom/HEDT PC market. Many expensive motherboards now come with a "dual BIOS" system that gives you an older known working image to boot from, in case flashing a new version broke something that can't be easily undone.

Another amazing feature is the ability to flash a BIOS from an unbootable system. You insert a flash drive with the firmware file into a USB port, press a hardware button and the BIOS gets updated, even without a CPU socketed.

This is a requirement for any motherboard I purchase now. I have enjoyed the ability to use AMD CPUs that are slightly outside of the generational support or enable features I am not promised.

Without the ability to flash from USB without a CPU doing this requires keeping spare CPUs that will work just to flash.

HEDT = High-End DeskTop, which (until 2022) referred to CPUs with more cores and separate sockets compared to ‘normal’ consumer CPUs, apparently.

https://tweakers.net/reviews/10334/het-einde-van-de-high-end... (Dutch)

As a user/customer, if I'm part of that 1% with an issue and get the same sort of "canned" response you see on the mentioned thread, I feel like me as a user doesn't matter. I guess the next step is calling customer support and then having the person on the phone making me go through their checklist of things I've already tried and again, feeling like this is of no use.

I think it usually takes a big rollout for these big companies to actually "hear" their users.

The second point is the really important one here. Mistakes happen, having a factory reset that actually works is crucial to avoiding extremely expensive recalls.

I'm reminded of the time a random NPR station accidentally bricked the infotainment systems on thousands of Mazdas and because there was no factory reset feature they had to spend millions replacing head units. That's just bad design.

Indeed a golden factory firmware version that will be booted automatically if all else fails and that provides minimum connectivity is crucial.

I wonder if that opens a threat vector from a security point of view? If an attacker knows that the golden firmware has some critical vulnerability which they can exploit easily, they can activate it at will by bricking the device and waiting for it to restart.

They could, and that's been a way for attackers to "jailbreak" devices and load custom firmware in the past. Though for the sake of reducing eWaste and enabling device repurposing and reuse, I do think this is the best path for firmware-updatable devices.

The golden firmware should reset to the old/first firmware of the device and nothing else. Keep it as simple as possible and restore the customer device back to an operational state.

The problem comes in if that old firmware has security holes, particularly if the device is network-connected.

The reset would be done physically. If there was some danger of the device being exploited after being reset, advice could be included for those performing the reset to prevent this. For example, to not connect it to a network and to manually perform an update to the latest version with some physical media.

Attackers aren't usually in a position to reset firmware, and if they are they might as well do a whole host of other things like replace the device with a compromised one. I don't think there is much of a point to trying to protect from that.

Ability to reset to original out of the box firmware is not only about failsafe. It's also a protection from "bug fixes" taking away features you had out of the box.

I'm still pissed off about LG removing record to disk option from our TV after an upgrade. I've only connected it to internet & upgraded assuming some of those bug fixes resolved few dlna issues otherwise it's always on internet block list.

> will be booted automatically if all else fails

I prefer to keep the factory firmware reset to a manual process that requires user intervention.

For example, holding down the reset button for 10 seconds after plugging the device in.

In my experience, it's not a good idea to have a device automatically roll back firmware and erase user data after failed boots. These mechanisms get triggered too easily during certain power outages (power comes on then goes off just long enough to cause multiple failed boots) or when users are doing simple things like rearranging their power cables.

The important feature here I would insist on is to let the user decide when to do a firmware update. Not the other way round. That's the way to build a good consumer relationship.

Why on earth a sound bar needs to update its firmware? Why firmware needs to be in a couple of tweeters and a woofer? It should basically output audio from an input source.

Another good one is; please always split any security updates from feature changes (and backport the updates per whatever versioning policy you have for those lagging the latest).

After many years of being burned I always delay system level non-security -related updates at least several days after launch to mitigate the risk.

> 2. A failsafe firmware reset back to factory state.

Do you mean like a physical button? That could work, though I'm not sure I've ever seen it. Holding down power for 10 seconds (or whatever) usually just erases user data, but doesn't reset firmware. Are you aware of any device that does this? But does it require some meta-firmware to roll back the firmware? What if that meta-firmware has a security flaw and needs to be updated? And that update is faulty?

If you're talking about a code sent from your servers to devices to reset, that seems like asking for the impossible. If a firmware update bricks the device, that may very well brick its ability to receive codes at all.

In both situations, it starts to feel like a problem of infinite regress...

Reverting to factory state seems riskier than last known good state. You could run into things like TLS root authorities not being recognised, deprecated cipher suites, etc. Just because that version worked a decade ago, it doesn’t mean it’s compatible with the world today.

> Reverting to factory state seems riskier than last known good state.

Reverting to factory state is the last resort. You don't have users do it unless there is no other good state to return to on the device.

> Just because that version worked a decade ago, it doesn’t mean it’s compatible with the world today.

That's why I said you have to include this in your test procedures.

When you're planning for the long term you can accommodate for these things on your servers.

> > Just because that version worked a decade ago, it doesn’t mean it’s compatible with the world today.

> That's why I said you have to include this in your test procedures.

You can’t test the world. Even if your servers can correctly respond to requests from old software, it doesn’t mean that the network between you will too.

Networking surely does introduce complications especially when TLS is now basically considered required and cert lifetimes are being limited for 'security' reasons. However most consumer devices have functionality, often their primary/most important function, to which network connectivity isn't even needed. For instance, a speaker producing sounds.

In the factory reset state, things should have a USB flash drive firmware install route which could be used to bring back working root certs, etc.

Of course again this depends on whether the mfg is worried about DRM bypass hacks that are found later on in the factory firmware.

I'd support legislation to issue stiff fines for devices that can't be factory reset at any time, with the only exception being for directly-consumer-benefitting anti-theft (so, iCloud lock is okay).

But can’t you? Sure, factory firmware from many years ago might have issues, but should still work well enough to allow you to fully offline upgrade to a newer working version.

I think all the OP was saying, is: Suppose you’re releasing firmware version N for some widget you make. Now, for all versions V in (0..N-1), verify that applying N to V works correctly.

> 2. A failsafe firmware reset back to factory state. Some sequence that resets the device completely back to the way it was when it came out of the box, firmware included, as a last resort.

That's a nifty mechanism that also allows downgrade attacks, so it has cybersecurity implications that may or may not be acceptable. Furthermore, it might not be practical or even be possible to restore the system to factory condition due to technical reasons.

The team next door allows its systems to downgrade to a previous minor version with a mandatory factory reset. It however refuses downgrading to a previous major version because it implies the bootloader was upgraded or the storage was repartitioned and they really don't want to rollback that.

Except when it comes to firmware, downgrade "attacks" are not attacks at all but just owners making use of THEIR devices. The real attack is the company trying to retain control over something they have sold.

But .. but then they can escape the extortion to a working state..

This is the de facto playbook for one of the Mega-Evil Corp.'s CPE firmware (Gateways, IPTV receivers, etc...).

New firmware is pushed in phases 1%, 5%, 10%, 25%, 50% then full scale.

Each stage has some delay incorporated for acquisition/application and then for telemetry (including support contacts from affected accounts) to determine impact and allow for regression fixes.

The other reason they would phase launches is because of firmware builds being used across multiple CPE models and hardware revisions, where only a small subset of hardware could wind up being problematic, but not discovered until deployment.

When you have millions of devices deployed, even a fraction of devices having an issue can create a shit storm on the support side of things.

It all seems so obvious once you know to think about it.

> "A failsafe firmware reset back to factory state"

A failsafe firmware reset back to a safe and secure state yes. The factory state is not necessarily that, so no.

I think devices should keep a last known good state firmware but keeping a full factory state immutable firmware would be irresponsible for many usecases.

What hardware reset typically does, in my experience, is to reinstall the last firmware you installed. Many don't even have the space to keep some original and/or safe image in addition. I'm working on one device where we delete much of the existing system to make space for even downloading a new firmware image. It's wild.

iirc for computers doesn't gigabyte have some kind of patent on dual bios design (active vs backup bios chips). I'm sure there are other ways to implement it but I think thats true.

I bet, but I'm talking about devices where the manufacturer tries to shave off every cent to price their products competitively. And then you have big meetings where you have to push back on storage being reduced by a further 2 MiB. At least that's something I've seen working in the embedded space. Storing an additional firmware image, be it only a few megabytes, is unfortunately often off the table there.

> 1. Staged rollout of firmware update

Especially if there is an internal testing stage before actually rolling out to production. It's possible that the users seeing the bricked devices are in fact limited to the initial wave, but the damage is already done.

> A failsafe firmware reset back to factory state.

Or perhaps to the very first released firmware version. This way they don't have to support updating from any version to the latest, just from the first one.

Also a dev or dogfood population of devices used by employees

Both are very reasonable features, of course. Here are (some of) the real-world challenges to their implementation:

#1: Requires competence, and/or management that isn't too focused on velocity and features to listen to their engineers' warnings about exactly the sort of problem being discussed here.

#2: Many firmware updates explicitly and specifically want to strip away features that the hardware shipped with (by introducing DRM, paywalls, etc.), so see the comment about management above.

Reply on news.ycombinator.com