Hacker News Reader

Comment by ajr0 | Hacker News Reader

> The Chief Data Officer (CDO) will "(1) be responsible for lifecycle data management"

I am very interested in what this type of lifecycle might look like considering most data I feel should be kept forever. I wonder how a lifecycle might collide with the challenges that bit rot[0] are dealing with.

[0] https://www.theguardian.com/technology/2015/feb/13/what-is-b...

View on HN

This is already getting really exciting. Some government entities (including lower levels like local, county, and state governments) have moved to digitizing their old paper and microfilm records. But if they're expected to maintain many types of records essentially forever, it places a constant burden to continue to update and migrate data in perpetuity, whereas paper or microfilm can sit in a box in a closet for decades.

For the most part, common file formats like PDFs, JPGs, and TIFs are likely to be understood for a very, very long time, but you don't just have file storage, you have systems to manage, index, and find those files, and those systems are likely to need constant maintenance.

It's not like paper in boxes don't have their own maintenance costs.

But different and arguably on-average cheaper costs.

Paper can rot if it's not kept climate-controlled, but not nearly as fast as bits stored to non-volatile media if the power goes out and there isn't a hot backup.

You can keep a whole warehouse worth of paper files on one drive though. 1TB is like 500M pages of text (1000 pallets).

Government records aren't "text" though. They're documents, which need to be preserved with any margin notes, markings, signatures, etc. intact. So if you're storing government records, you're storing images.

Physical storage space definitely does have a cost, but so does software licensing, hardware maintenance contracts, etc. And a lot of government organizations have physical space in abundance, but IT resources, less so.

Don’t know why this is downvoted. When a bank stores a record of a cheque it’s a scanned image for that very reason.

Oh, for sure, imagine trying to warehouse [images of] every submitted form in a condition that can be read - so, climate controlled at least.

I would think that outsourcing this to Amazon and Azure for redundant copies would be a sufficient start. AWS S3 and Azure Blob Storage would go a long way. Structured directories, document metadata complements and having hot indexes for said metadata in a database would be a very good start.

A lot of this type of work is already being/been done.

I mean, sure, until Amazon and Azure decide to delete it all [0] to save money, or those companies go bankrupt and shut off their servers. None of these big cloud providers is even 50 years old.

"Forever" is a very long time indeed. There's a reason the UK still keeps some records on scrolls of velum: https://www.bbc.co.uk/news/magazine-35569281

[0] - https://news.softpedia.com/news/All-Videos-on-Google-Video-W...

I doubt they'd both close shop at the same time... leaving enough time to shift the data to a new second source.

As an aside, nothing is forever... there are a lot of points in history where records are gone for ever due to fire.

Sure. If I were going for "forever" storage, I would use flame-resistant clay tablets https://www.theatlantic.com/technology/archive/2017/01/human... or perhaps a laser-etched nickel disc: http://rosettaproject.org/disk/concept/

But nothing digital, that's for sure.

Sure, but if you're legally responsible for holding onto records for a certain period of time ("forever" may be 80-100 years, realistically, in a lot of cases), you're expected to employ a reasonable practice to ensure that happens. In the case of digital files, it'd be crazy not to have an off-site backup for records you're legally obligated to retain. And if you do what is reasonable, and still the records are lost (such as to a fire), you're likely legally in the clear. Whereas if you destroy them prematurely with intent, or are even just negligent as to their maintenance or care, you may be in hot water.

I've seen Blu-ray disc manufacturers claim lifespans of over 100 years with capacities of 100 GB per disc. An entire warehouse of paper and microfilm documents would be able to fit in a shoebox.

Note that most record digitization projects are likely looking at live storage (online disk arrays), as it allows constant access to said records. Also note that the discs may last 100 years but having disc readers which can read them may not, and one would need to load all the discs to do a media conversion.

Reply on news.ycombinator.com