HP rushes to fix bricked printers after faulty firmware update | Hacker News Reader

HP rushes to fix bricked printers after faulty firmware update

bleepingcomputer.com

96 pointsmiles85 comments

Not so long ago, we advised the end user to patch everything as soon as possible, and depend on auto-patching as much as possible. The risk of their devices becoming hacked was much higher than the damage of software breaking, and big vendors worked on dependable patching.

Now the world evolved to the point that patches shamelessly remove features or install adware. Even the big names are incompetent enough to cause damage on a regular base. Meanwhile, the internet learned to deal with botnets as a fact of living.

So, assuming you live in a country where identity theft isn't much of a problem, and assuming regular and working backups are implemented, I start to wonder if it isn't time to review our best practices: Don't allow anything on the internet unless it really should (get a good enough firewall), don't run as root or administrator unless you have to, but also disable automatic updates, and do manual updates when a patch is out for 2 weeks and has proven not to cause more trouble than good.

So what's your opinion? How should a non-techy deal with today's landscape?

On every new windows 11 box of mine, I spend around 4-5 hours setting it up in a way to prevent any phone-home or auto-update. Group policies, deleting things, setting up fake domain controllers etc.

Preposterous amount of time spent to make sure I don't end up with "bing discover" feature that some product manager rammed through into the Edge or reset my settings because microsoft reaaaaly wants my new tab page for ad revenue. Oh look latest windows release has "microsoft rewards" in every frequently-viewed UX component; wonderful.

There's an army of CVE bros cargo-culting bullshit like "it doesn't work if it's not auto up to date", when the reality is the product doesn't work if the latest upgrade breaks my workflow or I have to stop what I'm doing and spend a bunch of time undoing whatever new "feature" got added for user-adverse but revenue positive purposes. I don't think I've seen an update in the last 5 or so years that didn't try to turn a thing that I own into a grocery checkout aisle for other stuff that I should own.

My HP printer sits behind a NAT and a firewall and the firmware has been feature-complete since they built the stupid thing. The only thing you get with upgrades is operational risk; hedging imaginary "someone's gonna p0wn it" problems is the security equivalent of the $5 wrench XKCD.

All that work to keep your own property from being weaponized against you.

Richard Stallman’s positions on software seem less extreme every year.

Ask 1990s or 2000s me, and I’d say he was interesting but “out there”.

Now, I’m thinking he was just 30-40 years ahead of his time.

Same. r/StallmanWasRight

I see the big fight now is over AI. The talk is over AI ethics and safety … but can we honestly entrust that to Big Tech?

Of course not. The unfortunate reality is on average the larger something is the less it sees you as a person. Any large organization sees us all as walking billfolds. If they don’t see us as human, how can they keep AI ethical or safe?

The reality is they simply cannot, and won’t. While safety will (probably) be assured, because unsafe products bring lawsuits, unethical products that extort customers for money are apparently perfectly legal. We see this in pharmaceuticals, processed foods, and of course Big Tech.

That leaked Google paper on how open-source models will catch up and surpass AIs from Big Tech is interesting. According to that paper, being able to train huge models is not a sufficient moat.

But requiring licenses to construct or deploy AIs would be a sufficiently large moat.

If you are asking yourself this question seriously, I implore you (whoever is reading this) to explore the incentives the different players have in every decision that is made.

The fight over ai is the same fight as it was before - user control and freedom over the ai

I should have control over model updates, retraining, the training pipeline, the prompts... Everything. How ethical or safe the ai is is my problem. It's not a matter of trust, but that it gets changed out from under me, breaking my workflows, and not letting me fix problems with it that affect me.

Would we have had GPT, et al. by 2023 without FOSS?

Unlikely, but there are other ways to control who gets to align AIs. Requiring licenses to construct or train models would be one of them. Patents are another.

No, and we can't trust it to big government either. This is my belief and fear, the government will or has already implanted itself with open AI exactly like they did with Twitter. So now we will have the most powerful corporation in the world secretly controlled by the US government.

Is there a lesser of two evils here? I honestly don't know, but I don't think we will get a choice.

I just wish he didn’t partake in pedophilia apologia

Maybe you can contribute your work to this existing project that is really quite good. For your consideration: https://github.com/builtbybel/BloatyNosy

> On every new windows 11 box of mine, I spend around 4-5 hours setting it up in a way to prevent any phone-home or auto-update. Group policies, deleting things, setting up fake domain controllers etc.

Would that even work on an Apple device?

>Would that even work on an Apple device?

Yupp and it's way easier to do - launchctl list and blow away what you don't want without the bs of looking at registry, then group policy, then domain controller nonsense, then services, then start up items then...

You just have to hope that Apple doesn't do what it's already tried and allow its own apps to bypass the TCP/IP stack filtering and firewalling.

And then backpedal furiously and say that it was "a temporary measure while some apps were getting fixes and updates". Huh - not sure why TextEdit.app ever needed a kernel extension but maybe they're right, and it wasn't just a BS excuse.

Why would you want to do that on an apple device? The point of buying apple stuff is that apple has full control over it

Because I own it. And the idea of apple being in control sounded nice in the beginning but wore off after about a week.

Why would you ever connect a printer to the internet? Best case it should be available on LAN, and you can use a VPN if you need a wider network (ie. an office building).

You should more or less _never_ update your 'dumb' devices unless there is a specific feature/bugfix you need.

A printer on a LAN will often find a way out to the internet. Sure, you can firewall it if you think to do that. Another way to auto-update firmware is that the printer driver on your computer fetches the upgrade (your PC can certainly get to the internet) and then installs it on the printer.

It's hard to stop sometimes. The printer driver might auto-update via windows, so a next print job auto-uploads new firmware.

The latest HP models require always-on internet or they will refuse to print.

I’m trying to work out what you mean.

Is your lan not connected to the internet?

With proper equipment and setup, you can control what on your LAN can talk to the outside world (you can get way more granular than that as well).

For all in ones that have a scanner, it's used to support scan to email without involving a PC.

I gotta tell you, I would be more worried about giving these things live email credentials than I would be about them spontaneously bricking.

Well you would be dumb to give it anything but unique per-device credentials to a working SSL-enabled SMTP server.

For non-techy home or SOHO users, they're likely using smtp.gmail.com with their gmail creds though. Would not be unresonable to dedicate a gmail account just for scan to email IMHO.

> For non-techy home or SOHO users, they're likely using smtp.gmail.com with their gmail creds though.

That's exactly what I'm worried about. Obviously there are ways to do it safely (and gmail actually I think might even force them), but I have very low expectations of a lot of the userbase (not a dig at them: the tech isn't exactly set up to make the easy thing safe).

I disable all auto updates on my mum's phones and my own as well - the pain of dealing with features being removed/moved/renamed, interfaces being redesigned and unexpected changes is far greater than the absolutely miniscule risk of being hacked. I hate with absolute passion how an "update" to my phone can change its entire interface to the point where I have to re-learn common usage patterns. Google apps especially are the worst at this. It should be exactly like the day I bought it, I used to think that OTA updates are the best thing ever - now I've grown to loathe them.

You think apps should never change their interface.. ever?

I think Google especially does it way too often. But I think to maybe make my argument more precise - I think it's fine if apps do iterative improvement of the interface. But what I see way too much of is "we've rolled out the new version of our interface, enjoy!" and they completely changed the theme, button positioning, names of functions, half of the old stuff is missing.....and that's just it, you have no recourse, no way to roll back other than installing dodgy APKs from somewhere. To give a simple example - TP-Link had a very well designed Kasa app for their smart devices, one day a new interface rolled out, breaking my presets, the interface is dog poop, some functionalty is actually missing - I sent them a long email explaining all of this, they just said "thanks, we'll think about it" and nothing happened.

So yes, in that case I'd rather that this app literally never ever changed - there was nothing wrong with it, but of course some designer somewhere has to justify their continious employment so they roll out these interface changes to basically have a job. I'm sick of it.

>assuming you live in a country where identity theft isn't much of a problem, and assuming regular and working backups are implemented

These are gigantic assumptions. I am not exaggerating when I say gigantic. Identity theft alone is hitting record highs in the EU in both percentage affected and money exploited.

The other problem is we tend to offer advice that is so vague it might as well be harmful. What does it mean for a patch to be proven to not cause more trouble? How does one prove a working backup? List goes on. We wind up giving so much of this nice sounding but useless advice that we confuse people. Even techies! You see it on HN all the time, bad/mediocre advice passed around and internalized, things regurgitated because it sounds good not because it's what will keep your data safe.

When advice is so horrible even techies wind up reverting to the most simplistic solution possible, that's a very clear sign we are doing a horrific job!

I'd like to hear more about what you know of EU identity theft. Specifically as I am from Belgium, my bank is protected with 2FA and my eID identity card should protect me from criminals creating contracts on my name. While probably not bulletproof, I can't really name recent common identity theft troubles. Most damage seems to happen by providing the android/apple/steam/... stores direct access to your bank account or identity, so AFAIK the main advice here is: don't.

Also: What is your alternative? I hate my own post above, but can't really think of anything better.

Without severe expansion of data privacy laws universally, most of my alternatives fall short. I can tell somebody to use 2FA and to be selective about their data, but what good does that do when companies are barely complying with GDPR as is? Our solutions are mostly to know bad things will happen and to have it affect as little as possible so you only have to change one password/account. Bleak. It's disappointing in a way, we could do so much better; substantially improved data privacy laws and mandatory e2e encryption would be great first steps, but they would enrage the very same corporations (like HP) that pull off this nonsensical garbage that puts us all at jeopardy. They keep pushing all these tracking services updates to those to restrict use and enforce sales, we can't handle this by just telling people to update less because eventually other companies will start adopting those policies if it results in increased ad revenue.

I also don't think your post is to be hated, it is not bad and is actually important in getting people on board with the idea of mass reinvention of digital services. We can't get there until we all accept that digital life currently is just far too invasive and we need better regulation before we can even start to work on personal advice. Such personal advice from Telekom[1] as an example, most of it sounds good but doesn't actually work in reality due to how weak our data privacy laws are.

[1] https://www.telekom.com/en/company/data-privacy-and-security...

I've always waited at least two weeks to a month before installing any windows update.

I've had two laptops bricked by faulty windows 10 updates in the past. I've gone through the whole gauntlet of people trying to gaslight me with statements such as "It is impossible for a windows update to brick your computer, or get stuck in an infinite updating loop."

I advise pretty much everyone I know to avoid updating important software automatically. Updates for critical software or hardware should only be done to a specific version that has been out for at least a month. Even then, always check multiple sources to ensure it won't introduce something unwieldy.

But of course, that gets a bit tedious for some people. So generally I only apply it to non-security updates, hoping and praying Microsoft hasn't screwed the pooch again with one of those. (Which has happened in the past.)

I never enable auto-update. It's just obviously a gaping vulnerability. It effectively delegates your security to a third party, so any vulnerability or incompetence in the third party becomes your vulnerability. You are effectively signing up to be a beta tester. No. Just, no.

My chrome instance complains about being not up to date because I rewrote the update daemon with a zero byte file that no one has write permissions to.

I did this after chrome’s auto update checker (not the actual update) pegged my cpu to 100% enough times that I rage deleted the updater. Then chrome kept rewriting it and undoing my settings to not update.

It’s a risk I run but chrome lost my trust.

Safari auto updates without ever causing me to notice.

> How should a non-techy deal with today's landscape?

Throw away any tool you do not know how to use. Get off the internet. Get off the computer. Revert 30 years (or more) of progress because we clearly went wrong somewhere.

> Don't allow anything on the internet unless it really should (get a good enough firewall)

Even that isn't enough, many a botnet has employed lateral movement inside a network.

>do manual updates when a patch is out for two weeks.

Manufacturers will simply ship dormant changes that wake up a month after deployed.

In some cases that'd be fine -- namely, if they used the extra time to extend the QA process. Of course we all know it wouldn't work that way.

I have no idea why anyone would buy an HP printer in the first place. The company is pretty much built entirely out of dark patterns ("You want to print a black and white document? Not while your magenta ink is low! Better buy a new ink pack. Or a new printer, they cost about the same.")

I've had much better experience with Brother.

I've had "better" experiences with Brother, but it still seems to suffer from being a printer in the ecosystem of "printer hell".

My Brother printer no matter what I do likes to go into this deep sleep mode that may as well be powered off, nothing wakes it up, power cycle and it drops right back into deep sleep immediately.... The software for it is bulky, wants to run on my PC all the time (why?).

Is it better than an HP printer, yeah probably in some sense, but the landscape of printers all being a pain is still true.

I don't have a functional printer right now. I need one ... but I keep procrastinating buying one. I just don't want to deal with it.

I have a Brother HL-L2350W that I paid $120 for, and is a basically perfect black and white printer for my needs. Plugged it in, set it up on Wi-Fi (LAN only). It goes into deep sleep after a few minutes of zero use, but even after weeks hitting print on any system in the house (no third-party drivers needed, at least on Mac) wakes it and the first page is out of the machine before I can walk over there.

I'm sure that, like any other company, they have better and worse products -- but I did very little research on this, bought to price point, and have been very happy. Certainly not the HP experience.

I'm a big fan of Brother printers. I've got a B&W multifunction one that's 10 years old with 50,000+ printed pages in still going strong. Our new color laser has been nothing but a dream to work with too but it's only about a year old.

If your printer is not waking up from deep sleep, I've found it due to poor multicast support on your network. I've had to debug various issues related to waking/discovery and once I resolved the multicast issues this problem went away. Specifically I ran into a UniFi bug with mesh networking and this problem is significantly exacerbated by poor mesh networking equipment.

Brother printers rarely, if ever, need the full software suite and they indeed have a "driver-only" install available most of the time.

Interested if you've got a write-up somewhere of the multicast issues you tracked down and fixed as I'm currently in a nearly-identical predicament with my home network, except I'm using mikroTik hardware instead of Unifi (ever since I mournfully retired my AirPort Extreme and Express's)

>My Brother printer no matter what I do likes to go into this deep sleep mode that may as well be powered off, nothing wakes it up, power cycle and it drops right back into deep sleep immediately....

Give the printer its own fixed IP address by specifying the MAC address in the router's DHCP table.

That doesn’t sound right. I have 2 brother printers at different locations and neither suffer from anything like that. One is a colour the other is a black and white. Both print perfectly with no issues so far. I wonder if you could update and reinstall the software to fix it.

Brother printer and no problems with black and white. Was curious about Xerox wax as that might have been better environmentally, not sure, couldn't justify colour printing costs though.

>Is your lan not connected to the internet?

Certainly not except for the few dedicated internet PC's handled by IT for email, browsing and company-wide office apps. Those have so many layers of hardware firewalls and other passive & active security it's not easy to get as much office work done as in earlier years, but we work around it.

In my chem lab with all those expensive (some vintage) scientific instruments, I air-gapped ASAP when I came on board. I have made lots of scientifc progress but the most valuable thing I've done for my employer is kicking IT out of the lab. 100x reliability after that, averaging less than a single need for attention per year.

Here's something I just found out recently.

Had one of the Windows 10 2019 vintage (installed from unpatched ISO) PC's on my isolated local XP network, where a very old Lexmark non-network USB/parallel printer on an XP workstation was just a regular USB printer being shared with the rest of the LAN by XP.

The Lexmark was a business machine installed with 32-bit XP drivers which were available in about 2016, which was about the latest it was supported, which support ended with Windows 7.

Luckily it turned out I had chosen the PCL6 version of the drivers.

When I had added the 2019 W10 PC I had naturally used a 32-bit version since this one does not need to use more than 3.2GB of memory at all. With the local LAN & W10 configured (not so easy) so W10 could easily access the Lexmark shared by XP, upon first connection W10 requests the Lexmark drivers before it will proceed. I used the W7 PCL6 drivers and everything was just fine for years.

Unfortunately the Lexmark tragically succumbed to an untimely death before it reached its 21st birthday.

Then I was able to select a consumer-grade new Brother USB/LAN/Wifi unit having a model No only available from Walmart (over the last few years since release) as the best bargain. Only an inconsequential distinguishing feature difference from the non-walmart generally available equal-printing-performance machine. It's another whole story but was out of stock for $89 with no sign of restocking, appearing discontinued and replaced by a new equivalent model for $139. I knew people must have bought quite a few at $89 before they were all "gone", and could see lots of them NIB for $139 on Amazon. I dug in and found a refurbished (designated 1 item remaining in stock) for $79 and it arrived shortly from walmart.com. Looked like a NIB return with a refurb sticker on the back. There was still "one" remaining in stock for weeks after that when I checked. Hmm.

Never uninstalled the Lexmark drivers, just plugged Brother into USB of the XP PC, used the Brother W7 driver, and worked like it should on XP, shared with the local LAN no differently than the Lexmark had been. Office manager and IT couldn't believe I got a brand new printer to just drop-in to my decades-old XP network when it was not officially supported for XP.

Newer W10 & W11 ISO's were used on a couple new local workstations but these were so new they were already sporting the "Print Nightmare" "upgrade" so no shared network printing for you. Oh, well. Turns out, naturally you can't use the printer's LAN connection simultaneously with its Wifi, but I wasn't actually using either, just USB. And the Wifi-Direct option worked just great[0] to connect to one of the new W10/W11 PC's on an individual ad-hoc basis as needed and everything was well from those wifi PC's and laptops when they needed to occasionally print. No Wifi-Direct conflict with the USB connection shared by XP over the local LAN at all.

Now I don't actually use the instrument the 2019 W10 PC is connected to very often and was occasionally printing from it over the shared XP LAN for a number of months before I recently remembered I had never installed any Brother printer drivers at all, never even made an explicit request to connect to the Brother on the shared LAN like I had to do when I first got the Lexmark going. Double-checked and nothing but Lexmark as default printer on that W10 PC.

Brother just worked. Really dropped right in in this regard.

PCL6 FTW.

[0]>likes to go into this deep sleep mode

If asleep, must be woken up from the main wired LAN the USB printer is sharing, before connecting a session of Wifi-Direct.

Edit: you may be able to set the sleep delay from the printer keyboard.

Too late to edit again, but wanted to add, in Linux Mint an older similar Brother at home just works when you plug it into the USB connection, alternatively on Wifi when part of the online LAN, just works when a Mint PC is on the same Wifi router.

No drivers necessary either way.

One more thing, when W11 was released, the Brother website said they would have W11 drivers within 60 days or so. Showed up early and were the exact same driver files used for W10 the last few years.

I heard Brother is following the same dark pattern.

Is the company aware of why they’re so loved/respected these days?

Do you have a source or a link? From my experience Brother is the best brand of printers that I have worked with (monochrome, laser).

https://news.ycombinator.com/item?id=31860131

Canon printers are the only ones that gave me peace of mind. I avoid everything from HP and everything from the top 3 brands as default.

I would love to know the source of this uttering.

At the risk of going against the grain here: I deliberately bought an HP laser printer and scanner combo(m148dw) a couple of years ago. Granted, the thing immediately got vlaned off from the Internet -- but that would apply to any device I don't fully control.

So far the performance has been rock-solid across several Linux and macOS machines and I almost forgot the saying "printers sense your fear and will break at the worst possible moment".

The newest consumer oriented HP printers (with model numbers ending in 'e') will not tolerate being disconnected from the internet. Even if you only ever print over USB, the printer has to maintain a wifi connection with internet access, or it will stop letting you print.

Out of idle curiosity I googled the current price for m148dw. I definitely don't think it justifies the current 400$+ price tag. Two years ago the prices were much more reasonable.

They are cheap. Most people don't realize the extra long-term cost. They only realize how hard they were fucked over when they need to get their first ink refill. I doubt many people buy more than a few printers in their life so the brand loyalty isn't strong.

I think the solution should be something like the EnerGuide labels that reveal the long-term cost. Then customers would be able to tell that this cheap printer sold at a loss will have crazy expensive ink.

Some years back I could usually find a cheap HP printer on sale for around $30. For the amount of printing I did, the starter ink would usually dry up before it was empty, and new ink cartridges would run over $40. It was more cost effective for me to throw away the printer and buy a new one instead of replacing the ink.

I bought one because I was ignorant and I just wanted a cheap printer. I mean, printing a paper, how problematic could it be?

I finally got fed up with it a few weeks ago and bought a Brother printer.

They're the reason why I sort my potential new printers by refill price before buying.

Because Wirecutter said a specific model was “less worse” than the rest of them and that’s all a lot of people really care to research.

Framework hinted several times that they could be doing printers [1]. That would be nice.

[1] https://twitter.com/search?q=printer+from%3AFrameworkPuter&f...

That looks like a potential overnight success. Everyone including my grandparents understand the problem with printers, inks and strange software they require to install, so the problem-space is well publicized.

The market for printers is dying though. These days I don’t know anyone who actually owns a printer. It just makes way more sense to use shared/commercial printing services.

Regardless of HP's actions to fix this, I'm curious what the law in the US and/or EU says about this.

If you have any kind of electronic device that gets bricked by the manufacturer via firmware update after warranty.

If the manufacturer decides to just do nothing (unlike HP which is trying to fix things here), what happens? Do customers have no recourse? Or do they sue the company individually? Or does it become a class-action suit? Or does it not go through the courts at all, but is there a regulatory body that fines the company for the value of the devices plus a punitive fine, and then distributes money or replacement devices to affected consumers?

Not asking what should be done morally, but what the law actually says?

I can see a bit of a problem with the whole idea of interfering with a good after it has been sold. It negates the original sale to some extent because the person that bough the goods did what they could to make an informed choice at the time and the manufacturer retro-actively changes the deal.

That negates basic contract law to a degree that I think that it already is illegal.

I mean, presumably if the actions of a company resulted in material damage (e.g. in this case losing a printer), you can sue them and the court then issue a judgement against them to compensate the losses. And if it happens to a lot of people at the same time, it can be a class action.

First they brick machines on purpose by requiring on-brand ink. Now this? I hope they are sued. Repeatedly.

No they didn't. I'm not defending HP at all, but let's stick to facts.

HP didn't brick any machines because of ink choice. They prevented them from working with off-brand ink, but the printers worked properly again once HP ink was inserted. "Bricking" is permanent, but the ink thing wasn't. "Bricking" was a disappointingly widespread misconception in the thread from 7 days ago:

https://news.ycombinator.com/item?id=35931468

Again, I'm not condoning the ink thing at all, but "bricking" is not what was happening. HP is already bad enough, we don't need to spread false information on top of it.

You buy a printer that you _know_ worked with offbrand ink when purchased (because it did), then one day HP remotely updates _your_ printer that you paid for and now it doesn't (and of course you are not allowed to revert the update), making it completely useless for the purpose it was purchased unless you go and pay HP for their extremely overpriced ink.

Your printer may work but now it's almost cheaper to throw it away and buy another one than refilling it. The only way it could be worse would be if HP decided to triple their ink price the same day.

If BMW updated and locked your car unless you used BMW branded wiper fluid (i.e. water) they'd be sued and scorned all the way to outer space, but somehow when a printer company does it it's okay.

I dunno, I think preventing it from functioning - even "temporarily" until the ransom is paid - qualifies as a soft brick.

There's no such thing as a "soft brick". To paraphrase one of the replies to the comment I linked to stated, calling something a "soft brick" makes as much sense as calling something a "soft pregnancy".

It either is or it isn't.

Bricking means that the device is as functional as a brick. Which to me means broken in a way the user does not know how to fix.

No point in suing them unless you have more cash for lawyers than HP does. And you probably don’t.

Surely that's what small claims is for?

I'm dealing with quite a few of these at work, HP claim they're going to fix it remotely but whenever I've seen this error before the printer is toast. (usually only about a month outside of the warranty period, make of that what you will)

They're not going to replace it because there's a fix coming "soon"

I've seen the same alert for a trio of HP printers on my network go up-down-up-down all week, really does flood your inbox after a while.

This might explain it.

I disabled auto-updates for my router. I also didn't bother setting a calendar reminder to check manually for updates and apply them. My router was compromised last week, 2 months after the latest firmware update was released.

Bummer. What did the compromise look like from your end?

Spewing weird and disturbing DNS queries. I only noticed because it was configured for NextDNS. Shodan tells me it was 53/udp dnsmasq, recursive.

A couple of months ago I changed the default gateway on my older HP Laserjet to some nonsense IP. Is that enough to prevent FW updates or can an update be “sent” thru Windows software/drivers or just over the network (eg without the printer itself having to do the downloading directly)

If it's connected to a windows machine there's a chance that they push out a firmware update through windows update, yes. It's happened before.

Reply on news.ycombinator.com