The looming software kill-switch lurking in aging Playstation hardware | Hacker News Reader

The looming software kill-switch lurking in aging Playstation hardware

arstechnica.com

137 pointsTomte81 comments

When I worked with CNC machines, one of the most interesting, pretty scary and surprising errors came from a dying CMOS battery. Having only ever dealt with such things in video game carts before, I was unpleasantly surprised when I learned the machines used battery powered RAM to store every single one of their internal settings, as in, if the battery died, it went back to even worse than factory default with literally no way to fix this once the battery died other than to have someone.from Minnesota come and recalibrate the entire machine.

We had about a week according to the error. We made a frantic search with various local suppliers, managed to secure a compatible battery, then I had to go on a hunt for the battery deep inside the console of the machine. Once I popped the battery out, I had two minutes to get the new one in before shit went FUBAR. It was a bit stressful.

Apparently the batteries had about a 7-8 year lifespan, the machine was about 8 years old. Nowhere in the maintenance manuals was it mentioned this may be a problem some day. The only info about this was an obscure error code.

Even the machine manufacturer technical support didn't know what was going on until they brought in one of the more experienced techs to talk to me.

I think it is most interesting that the PS3 was designed with the limitation of the CMOS battery in mind. As in, it was acceptable for the battery to run out in 15 years even with the network requirement. Maybe it was because the PS3 was one of the first consoles with a hard dependency on an internet service, and nobody bothered to look far enough into the future for the new implications. When someone thinks about a future where the PS6 exists, the conversation inevitably shifts back to the PS5, because the former is not perceived as an inevitability.

15 years, in the long-term span of things, doesn't seem like that long of a time period. It only sounds like it because we don't live that long, so 15 years is a significant portion of our lives, sometimes even a third of it or less.

Whenever an entertainer or someone reaches their N-year anniversary, I'm reminded that their career might not even last five more years, especially the ones whose youth or lack of it influences managerial decisions.

Opening Twitch, there were now zero streamers playing some games that were released only 6 years ago, even for big releases like the previous Smash Brothers. They still had a count of over 100,000 subscribers.

It really solidifies how transient a lot of human culture is. Games are shipped, money is made (or not), some of the titles are caught on the filter, the rest are forgotten.

Most likely, there were dozens of developers that poured out their heart and soul for some of those titles. Some of them might have had to suffer through crunch or the working environments of burakku kaisha with the cost of a lasting mental and/or physical toll. It's parts interesting and disheartening to think that some of their legacy will only ever have the lifespan of a CMOS battery.

In a world where emulation was illegal, I wonder if Sony would truly be comfortable with watching the last aging PS3 rust away and letting the console become nothing more than a footnote in the history books, one that future generations would only be able to imagine experiencing, despite the technological capability for preservation being right in front of them the whole time. Maybe that's how little value the experience as a whole would carry to them. People move on, and there are always shiny new ideas to think of.

I'm less familiar with the Sony side, but the NES and the original gameboy are still quite popular, which does make one wonder whether the newer stuff is purposely designed to be short-lived, or if there's a natural aversion to it --- if we're talking about entertainment in general, then even music several hundred years old is still enjoyed today.

Would it really be that costly for Sony to keep whatever NTP-esque server they have running? Sure, shut down the storefront, but an endpoint to check the time must cost peanuts for them to run....

The problem is it’s a tiny bit of maintenance that has to be justified for an unsupported system.

The better way is to factor out the time check/set and make it independent and identical across all systems - they they can close the older store but the newer one continues to reply correctly.

(Though it’s likely signed by certificates and will die when the PS3 root certs no longer get updates).

The flip side to discontinuing the old systems is regulatory scrutiny as they would essentially be stealing their customers purchases.

Or push a firmware update that removes that check.

The cost of doing that would likely be orders of magnitude more than just keeping an NTP server operating for a couple of decades.

No QA required: worst case is a different colored brick.

Does the PS3 work with IPv6? If so, you are probably right.

You mean commenting out a line of code, bumping the version and building a firmware image would cost a lot?

They should have thought about that and planned for it when designing the software

Then you need to keep the update server alive for a long time.

Or make the update available for download to install via usb drive.

Is that something that can be done now, or would that too require a firmware update to introduce that feature?

You can do it now:

https://www.playstation.com/en-us/support/hardware/ps3/syste...

But what about people who play time-limited PS3 games after their authorized expiration date! Won't someone think of the software companies?

I wonder how far out the expiration dates are? Surely they must all expire within a year or two? Can you get a time-limited game and say you want it for 50 years?

If Sony can't hold up their end of the deal with software license holders, then they need to work out a deal with the software company. It shouldn't involve end users at all.

It could be argued that this would be mandated by a right-to-repair law.

This was my question too.

I’ve never done it myself, but I’ve heard anecdotally that the Mac OS 9 ntp servers are still running...

It’s not the clock that’s the problem, it’s the license checks. I’m guessing here, but I imagine they’re tied to a time for validity, and the clock discrepancy triggers a remote verification.

> Sure, shut down the storefront

To be honest, I still don't understand why they needed to shut the storefront. They still need to host the games so people can access the titles they've paid for, and I would imagine that makes up the bulk of the cost.

The only reason that makes sense to me is that they have plans to shut the entire thing down, including access to purchased games, x years down the line. It's probably much easier to get away with that if even the newest purchase being removed is however many years old.

Not that that stopped Nintendo, they took everything down for the original Wii, purchases and all.

Thank God the internet never forgets. Wii games are awesome for small get togethers or rainy weekends. I'd pay a few hundred for a remade wii that has every wii game on it. Last I checked the vast majority of wii games clocks in under a tb, iirc. If I'm right storage wouldn't even be an issue as a 1tb sad is under $100, and the tech specs are equally light. The remaining problem is rights holders strangling the market to sell today's fidget spinner games.

“I’d pay ~$300 minus the cost of hardware in exchange for thousands of games, including timeless masterpieces like Metroid Prime and Mario Galaxy, many of which sold for $50 individually less than a decade ago and cost millions of dollars to develop.”

Don’t get me wrong, I’d buy that too.

But if it happened, I’d be more than a little scared about the viability of game development moving forward, particularly for the artfully-crafted single player experiences I enjoy and care about. More scared than I already am.

Nintendo would much rather port Mario Galaxy to their new hardware and sell it as part of a much smaller bundle. And as far as I’m concerned, that game is great and they have every right to continue charging top-dollar for it.

> To be honest, I still don't understand why they needed to shut the storefront.

I'd guess the cost of maintaining contracts to distribute licenses for games that publishers otherwise don't want to sell anymore.

> Not that that stopped Nintendo, they took everything down for the original Wii, purchases and all.

Yeah, this one was heart-breaking. A lot of my friends started getting real leery of digital purchases after the Wii debacle. At least the X360/PS3/Vita/Wii/WiiU have been cracked wide-open and the modding community can keep them going at this point - we've still got that to hold onto. The PS4, on the other hand...

The storefront was SSL2. It needed major software overhaul that wasn’t justified without being the star revenue source.

Yep. There is always plans to shut the whole thing down eventually, for console stores (the exception maybe being Microsoft?)

But you have to do it in phases to manage the negative feedback. If someone purchases a game on Day 1, you definitely don't want to kill download access on Day 2. So the only way to cleanly phase out is to first shut down any new purchases. Then you have a ~4-5 year window for people to play those games and hopefully give an upper bound for when they won't care about them anymore.

Sure, the article is about PS3 and PS4, but this scope really covers all digital "assets" that you "buy" (for various definitions of buy).

The more we see, purchases involving proprietary software/firmware are increasingly turning from actual purchases to horrific rental agreements. And those rentals aren't stated in clear language - you find out when you try to exert your rights... and you cannot. And many of these times, has arbitrary and pointless (for the human) internet "requirements" solely for the purpose of locking people they don't want out, and by tying people in a terrible forced ecosystem.

Until we have a buyer's rights of ownership with regards to embedded software, I cannot willingly purchase anything that requires forced internet, administered by myself, and/or completely software reloadable by myself or someone I hire.

And because of the extreme inherent controls that manufacturers try to retain control, that's why I recommend piracy. Piracy is the only real way to retain your money and retain control, cause I'm sure as hell not going to spend money on stuff that can be simply YOINKED away at their whim.

This is worse actually. Buying games on disc like any self respecting collector won't save you from this problem, because your console will stop working.

Anyway, Sony are complete idiots WRT to online. Back when the PS4 was new, setting it manually to use Google's DNS instead of whatever was built in improved download speeds by a large factor.

Also, when they launched in Europe they had PSN available for like 3 countries, so everyone else told them they're in the UK and called it a day. Now they have separate account systems for most regions, but the *holes don't allow you to change your existing account's region to your home country.

Could this be solved (hacked) with a MITM? Spoof the required responses to get the current time via raspi or something?

If you read the article you'd see that there's already a custom firmware that solves it.

With the caveat that you need to be running an old firmware to install the custom firmware, and there's no way to downgrade

The current last exploitable PS4 FW is 7.55 released in August 2020, if you updated to 8.00 or newer you're SOL for now

What prevents downgrading? Is simply a software issue or is there some hardware mechanism in place?

Don't know about the ps3 but the xbox one has efuses that get blown with most firmware update and prevent downgrading in any form, don't know whether any workaround has been found yet

The PS4 firmware is encrypted with a per-console key embedded in the CPU, so to downgrade you would need a copy of the old firmware taken from the same console.

Technically you can downgrade to exploitable firmware but only if the system had exploitable firmware in the first place, and had its firmware backed up, before being updated.

In some newer systems (the Nintendo Switch) there are hardware fuses they blow with each update. Their software won't work if the appropriate fuse is already blown.

That's not helpful. Yes I read the article, and yes I get there is a "solution", that wasn't my question. I want to know if the option I suggested is viable.

How hard is jailbreaking your PS3 nowadays? Asking here because I don't know which search results to trust across all the years of results that come up. A link to blessed, known-current instructions would be worth an upvote from me.

It's still my bluray player. I've never been too worried about it, but if it's going to half-brick itself it's hard to justify not jailbreaking it at this point.

The PS3 homebrew wiki on Reddit[1] is a good starting point. All the the tools you'll need can be found on the PS3Xploit team's website[2].

[1] https://www.reddit.com/r/ps3homebrew/wiki/how_to_hack

[2] http://ps3xploit.com

See here [0]. Generally on firmware 3.55 or older CFW can be installed directly. On consoles that came from factory with 3.55 or older but were updated CFW can be installed using PS3Xploit. Consoles that came with 3.56 or newer cannot be fully jailbroken, but some homebrew features can be utilized on OFW through PS3HEN [1]

[0] https://www.reddit.com/r/ps3homebrew/wiki/index [1] https://www.psx-place.com/threads/update-4-87-ps3hen-v3-0-2-...

This reminds me of some arcade games that had a "Suicide battery" onboard as a form of DRM.

A security key was embedded in these arcade board's processors which could correctly decode the rom. The key was backed by a battery. If the battery ran out, the key was lost forever so it was much harder for pirates to copy the rom.

I’d like to see an abandonware passed in the USA that says when a piece of hardware or software becomes abandoned and unsupported, the creator must open the system up completely.

And the unfortunate "solution" to that is the following.

Thing = $399

Plastic case of $thing = $1000

And I only wish I were kidding. I've seen plenty of things put out for washers, dryers, stoves, refrigerators, and vehicles that have that sort of insane "might as well buy it new" pricing schemes.

Planned obsolescence is one of those things I think future generations will look back on with regret, like dumping industrial waste into rivers and waterways (a common practice during the time period).

In a more niche area, some test equipment has this problem too, where the calibration data is stored in battery-backed RAM. To avoid having to do an expensive and difficult recalibration, changing the battery before it dies is done by either putting a new one in parallel temporarily, or with the unit powered on.

You’d figure equipment like this would simply have two coin cell batteries rather than one. It would make swapping them much easier.

On test equipment (and many older CRT projectors) these are typically lithium batteries embedded within the package of a non-volatile SRAM IC (often Dallas Semi) rather than an easily replaceable coin cell. There are zero exposed coin cells, not one, not two.

Capcom's CPS2 system is essentially the same, a battery keeps decryption keys in RAM, used to decrypt the game so it can be played. This used to make owning CPS2 games a real headache, but the community has now completely nullified the issue.

https://en.wikipedia.org/wiki/CP_System_II

I use FRAM for this exact reason. Higher cost but zero worries down the road.

We should thank Sony for creating a practical incentive for people to engage in the enjoyable hobby of reverse engineering, jail breaking and firmware hacking.

As an embedded hardware engineer working on yearly quantities in the low thousands who doesn't have to focus on unit cost quite as much as Sony, maybe a bit naive in this area? I've been using supercapacitors instead of batteries for my RTCs. Mainly to avoid service calls for dead batteries in hard-to-service products. I haven't done the math, but my engineering sense tells me that the added cost of the supercapacitor is more than evened out by the cost of batteries over the life of the product + service calls. Not quite applicable to consumer electronics, I suppose.

This smacks of poor coordination/planning rather than planned obsolescence, though for some "suits" the side effect is a beneficial one?

> that the added cost of the supercapacitor

Anything made in the millions of units per year is value engineered to death. Even if the cost is only $0.10 more that will really add up. I’m not sure the cost is offset when the repairs are out of warranty.

Which is so frustrating, as nobody would check the box for a 0.02% discount to avoid having to change batteries.

True, but it's the difference between making a single product that's identical and consistent, versus a made-to-order item. Inevitably when making a single high volume product, engineering decisions are made which reduce the envelope of theoretical potential capability compared made-to-order.

Right now I can go to the store and grab a console. If I'm not alright with a car on the lot of a dealer, I'm selecting exactly what I want, paying a bunch more, and waiting many weeks to take delivery. It's hugely more complicated from a manufacturing and supply chain standpoint - simply a different operating model.

I'd love to know who would willingly pay $234.65 for a console with "value engineed" nickle and diming rather than paying $238.87 for a few decent components

You don't decide to buy an xbox rather than a playstation because it's 5 bucks cheaper.

This applied to every single parts/designs so finally accumulated to dozens of dollars. IMO they shouldn't rely on external reliable clock.

I think that consoles (at least in first few hardware revision of given generation) are in somewhat special category. The hardware strongly looks like the engineering was primarily optimized optimized for time to market (see for example the amount of EMC shielding in both Xbox One and Series, which strongly suggest that the design target was to minimize the time spent on EMC (pre)conformance testing while ignoring the price of the thing).

I believe there are several consumer laptops such as MacBooks and Chromebooks now using supercapacitors instead of CMOS batteries

Super capacitors seem to die as well. I have 4 dead capacitors out of the 8 DS3231 RTC modules that I have bought within the last 2-3 years.

I think the original xbox used a super capacitor that has a very high leak rate for earlier models.

According to the specs, the supercaps may also fail within 10 to 20 years due to lifetime issues. They do not last forever and have a significantly lower lifetime than normal caps.

I've actually opted for batteries for very small currents (µA range) because the battery lifetime is about the same as supercap lifetime, so why bother with a charging circuit?

Two batteries would be nice. Or a battery plus a (normal) cap backup that lasts a week or so (or at least the time it takes to change the battery).

I had a phone system for a small office fail a few years back. We called a service company out and one of the first things he checked was a supercapacitor on one of the processing cards in the system that keeps some volatile memory stored through a power outage. Sure enough it had failed and in a few minutes he had it desoldered and replaced with a new supercap, he already had plenty on hand in the truck. I figured at the very least they would have swapped the card as a whole and maybe refurbish the failed one later on but due to the common failure of that supercapacitor they just always replaced it onsite. If that phone system had been designed to maintain that memory with a lithium primary battery instead it could have even been a user serviceable component.

What about LTO batteries? They have massive cycle lifetimes, and seem to not cause any issues if they fail.

>This smacks of poor coordination/planning rather than planned obsolescence,

I expect they get super-special company plaques at the awards ceremony for pulling a machine screw out of the design so 'beneficial' will be driven by a spreadsheet showing the warranty end-date.

I bought a PS3 to do Cell development and got the BSOD a little over a year later, it's practically the only electronics product I've had outright fail on me so Sony is still on my enemies list.

Guess it was a smart call for me to stick to playing Dance Dance Revolution on my old PS2 (unaffected by this and not connected to the internet).

Though I did have a time issue when I looked at the exercise feature, which started thinking it was back to the year 2000 (even though I had played in 2008 before taking a long break). That didn't otherwise affect gameplay though.

A similar problem is when Microsoft took down the update and activation servers for Windows XP. Hacks and phone activation sometimes work, but it is a problem for people who need XP.

"sometimes"? I think XP was the last Windows that has been cracked wide open (no pun intended), since the elliptic key they used was actually cracked.

Think I will shift all multiplatform console game purchases to Microsoft after this incident. They seem to have a reasonable backwards-compatibility story figured out.

At this point I'm pretty sure the only thing holding off OG Xbox and Xbox 360 compatibility on Xbox One / Series consoles is publisher approval. With the possible exception of Kinect games, I expect virtually every game on Xbox One and later to be supported as long as MS makes console hardware.

Wouldn't it be possible to replace the internal battery while bridging it with an external power source? How would the device know, that the battery was exchanged?

It's possible when the battery is still working. When it's dead, replacing it for new won't fix anything.

Yes, I understand. But I seriously consider doing this as a preemptive provision.

Did I read this right: This is an imaginary problem (doesn't exist now) that requires Sony to shut down a server they haven't shut down and didn't say they were going to shut down?

Sure lots of computers depend on lots of things. My windows update will probably cause trouble if Microsoft goes away. But its not really news, or unexpected.

I think it's the opposite.

I understand that if Sony go ahead with server shutdowns, there's a risk that when your PS3/4 battery fails and is replaced, you still won't be able to play games you've paid for offline.

Recently, Ubisoft has announced they'd be shutting down a couple of servers for older games.

https://www.pcinvasion.com/ubisoft-is-shutting-down-servers-...

https://forums.ubisoft.com/showthread.php/1241617-Online-Ser...

This danger isn't imaginary. I hope they push out a firmware update that makes it possible to ignore this kill switch should they ever decide to shut the servers down.

Certainly this is a feature and not a bug. If you can play games you paid for on a PS3 or PS4, then what's the motivation to buy a PS7 or PS8?

Which is why I only play games from GOG or games on my Game Cube and only drink pure grain alcohol and rainwater.

Did they confirm that this shutdown also includes that? It’s entirely possible they can float this endpoint on some current backend and this will just go retire on an island with the Y2K bug.

Does this issue also apply to Netflix and Amazon apps on PS3?

I'm wondering about this as well with the recent news of PSN's impending shutdown. I mainly use my PS3 for watching Netflix (besides occasionally playing Skyrim and Red Dead) and I'm wondering if I'll need to start using my HTPC for that instead.

I predict some day that we won't be able to buy clothes, strictly rent.

But there's probably a down-side in there somewhere.

Does it affect jailbroken devices?

Reply on news.ycombinator.com