Disc rot is a challenge for both archivists and collectors (2017)
vice.com
vice.com
Besides disc rot, optical drives are also at high risk of the laser dying - this has happened to a PS1, PS2 and Dreamcast of mine.
For these reasons, the most serious retro gamers I know are modding their optical drive systems to replace the CD drive with a SD-card based optical drive emulator.
Anyway, thank heavens for emulation and the homebrew fan community. Without them, so much of video game history would be lost.
Much of this content has popped up (or even been extended) on replacement trackers, but it's only a matter of time before the heroic operators of these trackers get forced to shut down too.
P2P could have made archivists jobs so much easier. This is a tech that should have been the next iteration of internet that we killed for no good reason. Not even greed, just a refusal to evolve a profitable but obsolete system.
For some reason the torrent community doesn’t have quite the same problem.
Other older platforms like Napster or DC++ (which actually still exists) let people share their own collection at the click of a button. This gave access to a really really long tail of rarities that perhaps only a handful of people were interested in, while torrent sites are mostly mainstream content.
What do you mean by that? You can upload your torrents to any of the big public torrent sites and if it's a private site just being a member allows you to upload already. Public trackers are also all using DHT which makes this not centralized.
Context: Torrent is hugely popular in Hungary, basically everyone is a member who uses the internet under ~40 years and is even remotely technical, which means more than a million people (with the single most popular site having over 700k members, with account sharing explicitly allowed). Perhaps not so much for the newest teenager generation (I don't know). Until very recently (and still now) streaming services have been limited contentwise and people just cannot afford to buy all the movies, songs, games etc. with the same ease as in Western Europe / North America.
Now that there are crowdfunding campaigns and tons of tools for microtransaction, that is even less excusable than when you had the effort to imagine the apparition of these obvious solutions.
It means we spend more effort on metadata (What are we allowed to do with it? Who owns it?) than the actual data.
It has stood directly in the way of new creative types (sampling in music, for example) and locked away content where the metadata is faulty/lost (orphan works problem)
It falls apart badly at scale (if you wanted to get distribution rights for everything on an OSX/Windows installation image, imagine the number of third parties alone, plus the original developers, you'd need to sign off)
and then it only really incentivizes work that's commercially appealing, not necessarily the technical best.
I dream of a day where people who want to create will be able to get a stipend-- it might cover a modest lifestyle, but ensures they can live the dream of being an artist/musician/programmer without having to clutch tight over every scrap of their output to keep their income streams alive. If we did it through a tax-funded endowment rather than market forces, we could probably finance a lot more creativity that way (think how many touring singers we could finance at 50k per year for the price of one Taylor Swift, or how many programmers we could hire to work om their passion projects in database theory for what is currently being spent on Oracle licenses.)
The only thing that gives me pause is that we transitioned to a system where individual music tracks are nearly free-on-demand* a decade ago and the sky never fell.
There were transition costs, and artists get way more from live performances now, but there's still an industry.
More than anything I'd just like to experiment with shorter copyright terms. I don't think the founders were crazy when they set them at around a generation. I agree that it might be risky to blow everything up, but I think there's a lot of bargaining we could do over term lengths that would be pretty harmless.
* To a first order approximation. Sure, many artists technically can get something when I pull up a track on YouTube. But if I'm not playing a whole playlist, I might not even see any ads.
What should we value more: hacker principles or phony outdated business models that aren't compatible with the realities of the digital age?
Copyright was created back when industrial capacity was required in order to make copies at scale. People had to own stuff like printing presses in order to infringe copyright at a scale that caused any damage. This is no longer the case. Copyright infringement is trivial now. It's as easy as copy paste. To solve this, the copyright industry intends to lock us out of computing. They think unlimited computers are too powerful for mere citizens like us and we should have access to nothing but limited versions that can execute nothing but predefined functions approved by them.
The mere existence of copyright is an existential threat to hackers and their values. If the preservation of free computing and internet requires sacrificing the entertainment industry, then so be it. The alternative is to sacrifice free computing.
Due to the existence of subversive technologies such as encryption, governments have been increasingly adopting the same perspective as the copyright industry.
The downside is that the post what community is so much smaller. Spotify et al really put a huge damper on the music tracker community, and that makes it less resilient.
People were saying the same thing in 2007 when OiNK shut down and What was starting up. The community is not smaller; it's more fragmented, and that's a good thing. Centralization is no better in our societal constructs than it is in our technological ones. I'll trade some immediacy and convenience for longevity any day.
I can only estimate because exact numbers are hard to find, but I'd say peak what era music tracker community total users was in the 350k region and we're probably somewhere around 100k now.
It does because private trackers use buffer as currency. The music trackers have had to adapt to use points systems now to encourage downloading activity. But even still, activity is way down from the what.cd days. It's just easier to stream the easily accessible stuff.
People didn't go to what.cd and jump through all those hoops to find rare music, they went to find good music, and now Spotify and YouTube do a good enough job of that with a lot less effort.
As a funny example of that I discovered Billie Eilish when Ocean Eyes first came out from a private tracker and now she's about as big as anyone. So it isn't just about discovering rare czech folk singers, but any music that you might like.
I think the hardcore demographic you mentioned is spot on and does exist, but they were always a minority and the evaporation of the less hardcore users explains why the scene is so much smaller now.
it was that process that helped to curate one of the highest quality audio libraries to have ever been.
it's not all that important for most folks to listen to and compare 20 different releases of Bowie's 'Rise and Fall of Ziggy Stardust and the Spiders from Mars', but to those of us that enjoy music cataloguing as a hobby , the loss of what.cd was pretty terrible.
https://3.14.by/en/read/cd-dvd-microscope
https://www.youtube.com/watch?v=Y8_HCLPdpkI (CD-R)
It would be interesting to see what "disc rot" looks like at that level of detail; and if better reading hardware may be able to read it. In particular, pressed discs actually store data in the varying height of the polycarbonate disc, so as long as those pits and lands haven't been damaged, even if the metallisation is gone, the data remains intact; and if there was a way to strip and re-coat that layer, pressed discs could be recovered. For CD-Rs, perhaps the dye has faded to the point where the typical drive (which can't really spin the disc slower than 1x) won't be able to discern the pits from the lands, but if that data can still be seen somehow, it's recoverable.
Now my hardware is better, CD and DVD are easy to observe without physical damage. I also can see data on BD disk, but it is on the border of what is possible.
I also tried to decode data from the image, but it will require more collaboration.
Azure apparently has an "archive" tier for 0.1c, which you can lower even further with long-term high-volume reservation: if you reserve 1PB for 3 years it's 0.081c/GB/mo (using only local redundancy) in the cheapest DCs where it's available (some DCs either don't have that feature at all, or it's way more expensive e.g. 0.1636 in asian DCs where it's available).
[0] https://cloud.google.com/storage/pricing#storage-pricing
ISO cert' states 1000 years mean lifetime (coin flip chance) but only 530 at 95% confidence. And that's at a constant 22C and 50% humidity.
And that's solely for the original 4.7GB DVD, not blu-rays.
> There have even been some accelerated aging tests that were able to somewhat confirm that.
OTOH others (the French National Laboratory of Metrology and Testing) did not and found m-discs to be no better (and possibly worse) than other high-quality discs with an accelerated aging setup of 90C 85% humidity.
Cartridges are generally more hardened unless they contain some battery-based saving system and even then a dead battery can be replaced.
The majority of them developed white spots or detachment of the disc layer. This has been known since the early 2000s, I’m surprised anyone would collect them without being aware of the risks.
(apparently defunct) syylex's GMD would most likely be structurally immune to disc rot, but you had / have to really really want to make your data rotproof: single-disc (4.7GB) order was 160€ (>$200 at the time). Not including VAT/GST.
No way:
https://en.wikipedia.org/wiki/VisiCalc
VisiCalc: Initial release 1979
https://en.wikipedia.org/wiki/SuperCalc
SuperCalc: Initial release 1980
https://en.wikipedia.org/wiki/Multiplan
Multiplan: Initial release 1982
https://en.wikipedia.org/wiki/Lotus_1-2-3
Lotus 1 2 3: Initial release 1983
https://en.wikipedia.org/wiki/Boeing_Calc
Boeing Calc: Initial release 1985 (6 years after VisiCalc).
As it was released, it could read "standard" VisiCalc and Lotus 1-2-3 documents, according to Computerworld:
https://books.google.at/books?id=97eagSKygYsC&pg=PA48&redir_...
What's interesting about Boeing Calc is that it allowed storage of its documents "online". But it was surely only for specific local setup, so it was probably "on the local server."
There's par2 (and its offsprings MultiPar and QuickPar), there's rsbep, there's pyFileFixity.
All have their problems why you probably won't "shield" your files with them (I don't, although I worry).
You'd think there'd be a market for long life HDDs.
Edit: Yes, you could just replace old hardware with new hardware, but it creates so much waste...
It apparently uses a similar basic process as those mall kiosks that take a volumetric picture of your face and then etch it into a cube of glass?
But in this case, they just write the data to the side of a small piece of glass over the course of days and it's basically permanent for all eternity unless something breaks it.
In practice, hardly anyone uses up the entire write endurance of their SSD, especially not if they're using it to store backups, archives, or other relatively static data. Flash memory that isn't worn out has much longer retention times.
Your situation is likely happened because of bad RAM.
I don't see how that could have happened, since the image wasn't ever written to. It should have been the same file on disk as the backup, since I hadn't touched it since backing up the file, but it wasn't.
> You don't need a serious amount of space, for example ECC RAM uses one parity bit for 8 bits to fix one bit flip or report 2 bit flips
The thing about parity is that for it to be useful in a data recovery scheme, you have to know what bit got flipped. That works for RAID of course, because you usually know which disk is bad and so when you replace it you can work out the missing bits using the parity disk, but I don't think it could work for an individual hard drive, since in general you don't know which bit flipped.
And the thing about checksums is that they can tell you if your data has become corrupt, but they can't be used to fix things behind the scenes.
I've been wanting to try to understand them to see if I could implement something myself for a while now
Reed-Solomon is often used in storage because it is an optimal erasure code - e.g., I know this block is missing or corrupt, correct it.
I was initially thinking OP might be talking about something like this, but I think I would have heard if all new hard drives had it built into their firmware to store error correcting codes alongside the real data and automatically fix bit flips, since I suspect that would have pretty serious performance impacts in certain cases and people would be complaining.
The entire photography community begs to differ.
Gradual degradation of stored image data across HDDs of all vendors is very well documented, mainly because the bit rot is very easy to spot when working with visual data.
Bit flips are persistent and they occur on the media itself, not in transmission. There are plenty of theories, including one that argues that bits are flipped by stray neutrinos, but nobody knows for sure.
[1] https://arstechnica.com/information-technology/2014/01/bitro...
(edited to make more readable)
Also ZFS. Part of its purpose and design goals was countering bit rot (as well as device hostility to keeping data safe), as Sun had customers affected (even with ECC). Hence the end-to-end checksumming amongst other features.
scrub exists pretty much just for bit rot: you run scrub regularly, it goes over the disk, checks that every block checksums properly, and if they don't[0] it repairs the data using the non-corrupted copy (assuming you have one).
[0] and your dataset is replicated, note that even if you don't use a raid configuration you can mark important dataset as to-duplicate for this purpose (this is not equivalent to device redundancy, it's a feature which exists solely for bitrot / corruption protection)
This is not correct. All forms of Flash memory lose data (charge in cells) over time. Even glass window EPROMs are only rated at tens of years. Every new generation of SSDs uses smaller Cells that lose charge faster. Wear and tear accelerates this. Samsung 840 with 300TB wear will lose data when unplugged for a week. https://techreport.com/review/25681/the-ssd-endurance-experi...
Its much worse in enterprise settings: https://www.extremetech.com/computing/205382-ssds-can-lose-d...
It also seems to be available on Sourceforge [2], but that seems to be about 5 years behind what was on the now non-existent dvdisaster.net website.
[1] https://web.archive.org/web/20180428070843/http://dvdisaster...
Has anyone seen a list of when disc-based systems might start showing signs? Something like game consoles and capacitors.
On a serious note, I’m in a similar boat, we’re in a middle of the move and it appears that my cd collection is in same predicament. So I’m debating whether to throw them to trash, or keep around (mostly for cover artwork, and to keep my Exact Audio Copy -> flac transfers just a bit more legit (I care of such things)).
Tidal goes higher, to 96khz on master quality where available (the vast number of tracks are not master quality, sadly).
More then that I have an old box of 3" floppies dated 92-95. I recently decided to check some old code, bought USB floppy drive and I was actually able to read them. Well not all of them but still ...
Nearly all may = massive decline soon as much as may mean you are done for another decade...
If you value it, move it now.
As for those floppies I was just curious. I could care less if I did not find that old piece of code I mentioned. I was just curious to look at it and see how my way of thinking did change.