Ironically my experience has been exactly the opposite. It's the demosaic-ed, fully developed copies of my photos that are larger and harder to preserve than the original RAWs. And these files can't be trivially regenerated from the RAWs, either. Let me explain.
The most important issue is that the process of taking a raw image and turning it into an edited RGB-pixel image is not obvious, at all. Tons of steps have to happen in this process, and there's currently no way to describe this process in a way that's compatible with a single open standard or even multiple pieces of software. At all. The steps can be broken down roughly into a series of "instructions", but what those instructions mean (e.g. to Lightroom) is entirely opaque and even secret in the case of closed source programs. Even open source programs, like RawTherapee and Darktable, use entirely different and incompatible approaches, algorithms, and instructions.
What this means on a practical level is that you can preserve your raw files and the instructions for replicating the edits as carefully as you like, but without the exact same piece of software (often even the same version of the program with the same settings and defaults set), your edits are as good as gone forever.
As a result I've had to take fairly drastic steps to make sure my photos are safe. For my older photos, I keep an entire VirtualBox image with Windows and Lightroom backed up along with my raw images, so that I can be sure of restoring exactly the same output files if necessary. (This more-or-less has to be a hacked version of Lightroom because you can't take chances with licensing problems preventing the program from running now that Lightroom is subscription based.) And actually for newer photos, I've moved away from Lightroom to RawTherapee. Even though I feel it's usually inferior, I feel safer since I know the pipeline from raw to burned edit is essentially public. I keep a backed up copy of the RawTherapee source code, but even if that somehow failed someone could make a RawTherapee compatible raw converter from scratch.
So that's why edited images are actually harder to preserve: whereas with raw you can just save them in 2-3 different physical locations and storage devices, to keep the edits you have to take several additional steps and there are more points of failure. Why not just keep the static edited images too? Well, I do that for my most essential photos. But that gets into the other issue, that the edited images are actually larger than the original raws.
If the point is preserving my edits, not just having a copy that's good enough for Facebook, the images have to be lossless. This pretty much means PNG or TIFF, and in fact the latter seems to be necessary since PNG doesn't handle metadata anywhere nearly as well. Unfortunately, while the compression used on raw files tends to be pretty good (which is further aided by the images being mosaiced), the compression algorithms compatible with TIFF are pretty terrible. Add that to the fact that you almost certainly want 16 bit images, in order to preserve as much raw detail as possible (in case you want high quality prints or need to do further editing), and you end up with whopping huge TIFFs. I regularly see my TIFF output files 4-5 times larger than the corresponding raws.
In short, managing backups for a high quality photography workflow is actually a good bit more difficult than it seems at first sight - and how it looks at first is not that easy either!
In one case you've lost original sensor information, in the other case you've lost some parameters and possibly the algorithm used. The former is infinitely more difficult to synthesize from scratch.
I'm only a hobbyist photog and the bday parties and weddings I've shot result in thousands of images. I can't imagine having to shoot multiple events every week for years. For a pro, the edits represent thousands of hours of work. That the work can be redone theoretically might mean very little to a working photographer.
This is exactly the issue I had in mind, well said.
Raw images seem to be very important until the very moment they’ve been edited (or, hell, rejected from editing). Then their worth is very little.
Raw files, and sidecars, retain their value because sometimes one wishes to revisit an image/edit. If I had to pick one or the other, I might prefer to lose the edited images. That is because I believe I have more/better days of my editing life ahead of me and because I underestimate how much editing I have done. Moreover, there are plenty of RAW images awaiting the day they come to life.
In commercial work, like weddings, the final product is the edit. If there is one thing that must be retained, that is it.
File sizes (DNG lossless, other formats 16 bits):
* DNG: 18.5 MiB
* TIFF: 84.2 MiB
* FLIF: 55.1 MiB
So while FLIF is unsurprisingly much better than TIFF, it's still ~3 times larger than the DNG, which means that a pretty noticeable amount of additional disk space would have to be sacrificed to store the final edited versions of all my photos.
`git checkout v3.0.2`
`./build.sh`
No idea what you're talking about. As long as you have your RAWs and your sidecar file you can trivially reproduce the picture, at least that's how it works in darktable.
Hell, the old code that is deprecated and "hidden" in the newest versions is actually all still there and if you import your stuff in the newest version it is practically guaranteed to look exactly the same. In fact there is a CI system that checks for delta-Es in each release.
> the old code that is deprecated and "hidden" in the newest versions is actually all still there and if you import your stuff in the newest version
I'd expect this to be the case with any good photo developer, but it doesn't pay to take chances. There have been significant bugs in the past with DNG files, for example, and different bugs could be introduced in the future, or bugs you didn't know about could be fixed, and so on. There's a lot of stuff that could go wrong. It's good to be able to just tar the source code.
I'll be honest, this is probably the most ridiculous argument against anything I've ever heard on hacker news. This is like saying C/gcc sucks because Javascript exists. What the fuck?
> I'd expect this to be the case with any good photo developer, but it doesn't pay to take chances. There have been significant bugs in the past with DNG files, for example, and different bugs could be introduced in the future, or bugs you didn't know about could be fixed, and so on. There's a lot of stuff that could go wrong. It's good to be able to just tar the source code.
So... it's a non issue? I don't even...
Well, sorry you feel that way. /s
The difference is that C and Javascript are widely implemented open standards. Lightroom, Darktable and RawTherapee are using three different opaque, undocumented approaches to developing raw images. Neither one has been reimplemented in other software, and it would be extraordinarily difficult to actually do that, because of all the specificity and quirks. You basically need the original software, which means making sure you can still compile it, making sure you have a platform it can run on, and so on. This is more complexity than most people ordinarily expect when they talk about "backing up photos", and that's exactly my point.
> So... it's a non issue? I don't even...
I should probably not even bother responding, since you already showed with your original comment that you didn't bother to even read my post, but I really don't get this. I explained a very specific issue: that there's a lot of complexity to developing raw photos, which means you have to take extra steps to make sure your edits are properly backed up. Open source software has a distinct advantage because you can archive a copy of the software yourself, but it doesn't change the fact that you do need the original software. And in fact you might need the same version, that's what the example of DNG conversion is supposed to illustrate.
This is ... the opposite of a non-issue. In fact it's a very specific issue that I took quite a bit of time to explain in detail. Having to back up your software, or possibly even an entire VM along with your photos to make sure your edits are preserved, goes way beyond what your average person, or even many photographers think they have to do to keep proper backups.
Why is this a problem? That's like saying "I need a good GPU to run this game, this sucks!" Well, no shit I guess?
And it's not even close to that because you can always just buy/compile/compile the right version. And for darktable you don't even need that, backward compatibility is guaranteed. Also darktable has a rudimentary Lightroom to darktable conversion tool (never used it).
> This is more complexity than most people ordinarily expect when they talk about "backing up photos", and that's exactly my point.
Not sure about the other ones, but for darktable, it's as simple as keeping the XML alive.
> I explained a very specific issue: that there's a lot of complexity to developing raw photos, which means you have to take extra steps to make sure your edits are properly backed up. Open source software has a distinct advantage because you can archive a copy of the software yourself, but it doesn't change the fact that you do need the original software. And in fact you might need the same version, that's what the example of DNG conversion is supposed to illustrate.
If you don't like having a non-destructive copy, save a high quality JPEG with your edits. Also, I can't think of any other applications (not just photo editing) that you can easily have future-proof non destructive copies of whatever you need to save.
For example: for audio, you can always just save a WAV or FLAC of your work, but you're still relying on the fact that your DAW workflow is still gonna exist 10 years in the future if you try to save your project as opposed to a mastered copy.
Code is also similar. You can save a binary which will probably work, but if you want to save your repo, unless it has good package managers you might have issues later down the line, like that left-pad incident a while back.
Hell, even for something completely different: language drifts over time, and reading Shakespeare is kind of hard unless you know how to read Middle English. And we don't seem to have a problem with forward compatibility, it's just a bit of a pain in the ass. I read it in High School and I didn't go insane.
This isn't a photography issue, this isn't a software issue, this is a core part of human experience.
> This is ... the opposite of a non-issue. In fact it's a very specific issue that I took quite a bit of time to explain in detail. Having to back up your software, or possibly even an entire VM along with your photos to make sure your edits are preserved, goes way beyond what your average person, or even many photographers think they have to do to keep proper backups.
You don't need a VM, at least for darktable. You're making an issue that hasn't appeared but is potentially possible into a real issue for yourself. Maybe it will become an issue, but just figuring out how to install the old version is gonna be good enough. You do you mane. I'll stick with running dt natively and trusting the forward compatibility and OpenCL acceleration.
(“Replication isn’t backup” still applies, of course.)
Backing up databases (terabytes) is feasible, they're not that big.
But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive.
I am curious, though -- for services like Dropbox or Google Drive, how many replicas are there of your files? I know there must be redundancy in case a disk fails, but do they keep 2 instances of your data or more? And are the instances spread out geographically, some or all?
Yes - I work at one of the FAANGs for a team that's doing precisely this. We develops an internal disaster recovery tool that creates backups of data files that can't be touched by the creating application, and that can be read back in a disaster event to recover the data.
I don't see the size as the big problem. If one cloud can handle the size, the backup storage system can too. For me, the real issue is timing. How do you backup a cloud full of constantly changing data? Do you draw a line in the sand, an image of the cloud state at a particular moment? Is that even possible? You have to do backups of smaller chunks, individual accounts, but eventually that just looks like another software-managed internal array structure rather than a true duplicate. Your backup system is just as susceptible to deletion error as the cloud it lives within.
Mandatory xkcd: https://xkcd.com/1737/
If you even do the straightforward "don't freeze time anywhere, just rolling copy all the files onto a hard medium", you may end up with logically inconsistent data, in that files from later in the backup may actually not have existed alongside ones from earlier ones, but if you write down the timestamps periodically, you end up with a way better end state than "I lost all the files".
At worst, you end up with "I may have lost the last 3 minutes worth of files from the last 3 minutes of the last backup I did".
I can only describe what we do, and, of course, there is an enormous scale difference here, but ...
Every single rsync.net account is it's own ZFS filesystem which means every single account gets snapshotted[1] nightly on a schedule. This means that the enormous operation of "backing up" all of rsync.net happens in small, manageable, chunks.
Of course, the ZFS snapshots of a customer account are not "backups" per se, but if a customer choose "geo-redundant" storage in another facility, it is those very same snapshots that we (zfs) send over. Those are, indeed, backups.[2]
The most interesting part, in my opinion, is that the daily/weekly/monthly snapshots are immutable. So you can publish your rsync.net credentials or suffer a disgruntled employee or ransomware attack, etc., and those snapshots remain safe - they are read-only.
[1] Accessible, browseable, in ~/.zfs/snapshot
[2] GR storage costs 1.75x normal pricing.
It could be argued that one needs to do a site wide "write cache flush", stop, and snapshot. Not that I think for one second the service provider should (or could) be in a position to detect when a good time to snapshot might be....
Any serious database does offer backups in addition to replication, and that kind of validates OP's point. As for object stores there are a variety of techniques [0].
> But an entire cloud storage for photo and video for millions, we're talking maybe exabytes. The notion of making "separate" backups seems cost-prohibitive.
Cold-stores like the one Facebook built would be apt [1].
> I know there must be redundancy in case a disk fails, but do they keep 2 instances of your data or more? And are the instances spread out geographically, some or all?
See [0]. Backblaze do share quite a bit too about the space, too [2][3]. With minio, one could run their S3-like system on-premise [4].
[0] https://maisonbisson.com/post/object-storage-prior-art-and-l...
[1] https://engineering.fb.com/core-data/under-the-hood-facebook...
[2] https://news.ycombinator.com/item?id=10540361
There is also the option of using Glacier for lower cost long term storage.
If you are asking does regular S3 provide a true backup capability out of the box, it does not, or at least did not last time I actively looked at this about 2 years ago.
Judging those costs and making that call is a complex matter of course.
(Honest question, not provocation. To me either you trust AWS or any party to never lose your data [unwise], or you basically ask them to offer a way to rsync, which AWS does)
designed to provide 99.999999999% durability of objects over a given year
https://landing.google.com/sre/sre-book/chapters/data-integr...
The short version is that the backups are a mix of short-term local backups + backups across the network on distributed filesystems + offline tape backup.
I can't speak to Drive, but the tape backup is certainly used for GMail. (There's a case study about Gmail having to restore from backup in the above link.)
I had a client where a glitch in the Dropbox sync caused 300k+ files to be deleted when we added a new PC. Dropbox support was unable to successfully undo all the file changes, and I had to get a ticket with CS escalated to a special team to get everything restored. Even when it was finished, they could not give a guarantee that every single file deleted was restored.
Maybe things got better since they've launched Dropbox Rewind, but given that revision history has been a feature for years and it still didn't work right, I no longer trust them as a backup.
We do.[1]
For exactly 1.75x our normal pricing, we will replicate your entire account, nightly, to a geo-redundant site which is not open for normal customer use (and, therefore, has a lower risk profile). This GR site is the he.net core datacenter, in Fremont.
It's also worth pointing out that replication to rsync.net buys you malware/ransomware protection since your account is snapshotted, by ZFS, nightly, and those snapshots are immutable (read-only).
[1] rsync.net
But they pay for it.
Ideally, a "delete" should mark images as unavailable and queue them for deletion at a later date (e.g. in 30 days). This provides protection against accidental deletion by users, accidental account deletions/deactivations, paid accounts terminated due to lack of payment, automated software mistakes (such as this), and so on.
The last company I worked at, 80% of our database was inactive, useless, or redundant data, all kept to protect against all kinds of issues (such as reused usernames, subpoenas, annual reporting, etc). We could always zero out PII from an account, but we never removed a record we didn't have to. It led to a lot more RAID migrations than I would have liked to have done, but it certainly made the application easier to manage. Plus, we never worried about fragmentation or holes in our data files, cascading deletes, etc.
One computer had a hard drive failure, BTSync deleted all the files from the computer that didn't.
Doubleplus ungood.
It’s important from a backup perspective that it’s point in time as well, otherwise as soon as you get ransomware that encrypts your file you now have replicated those changes everywhere.
[1] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
Given the peeks that amazon has provided into the scale of S3, I don't know if you CAN 'back it up'.
Allegedly, compliance mode is unalterable by any account period, I guess the equivalent of the immutable attribute without an overriding account. I'm not immediately familiar with any literature on attacks on this feature, but I've also not searched hard; however, I know from my clients that it's an accepted form of WORM, and that cloud storages like S3 are considered in the same vein as tape when immutability is in play.
I suppose it will be a case in the future that proves the efficacy of AWS/Azure/s3 providers, but for now, a lot of regulatory policies for 3-2-1 allow for such storage to fulfill the "2" part of the 3-2-1 rule.
A slightly larger company also used tapes, and stored them at an undisclosed (to me) offsite location.
A much larger company kept it all in datacenters, but the "offline" backups were disconnected from the WAN when they weren't actively being used.
Depending on how much data you're backing up, sneakernet works.
When I was still in the office, my company had me rotate a set of backup hard drives between the office lockup and a strongbox in my house. The notion was that it was unlikely that both the office building and my house 12 miles away would both burn down at the same time.
Of course, now I'm working from home, so all of the eggs are in one basket again.
1) That nobody in our company actually knew where they were physically stored (insider risk?), but
2) That we had assurance that the physical storage was far enough away that a physical disaster in our area wouldn't also touch where the offline storage was located.
There's a concomitant jump in RTO if that's what you do this, but hopefully that's well understood among the stakeholders.
The drives are encrypted (Truecrypt) since they will be outside my physical control. The ones at the office I am prepared to abandon should I get fired/laid-off.
Replication usually involves storage that is powered up and connected somehow to the same systems as the main storage; and any data that's corrupted on the main storage would propagate to the replicated storage.
In short, if anything that happens to the files is immediately copied to the "backup" then you don't actually have a chance to recover from any software problems. Whereas, if you make a copy of the data every night and keep the last 30 copies you can find an issue like this and go back in time to retrieve the files from before it started.
A backup can be a 1:1 copy but it should be set up such that something going wrong with the primary (e.g. cryptolocker malware), can't affect it (since "something went wrong with the primary" is what the backup is intended to resolve).
To do that you could take the backup offline or use features like filesystem snapshots to ensure that changes can be rolled back.
If you `sudo rm -rf / --no-preserve-root` your drive, replication deletes everything, while backups let you restore to the last time you took a backup.
Backups are WRITE ONCE. They persist if the original is deleted or modified.
Separate infrastructure and geographical distribution are orthogonal.
Imagine if you had a document that stored useful information. You had this document automatically replicated to another system in a different time zone every time there was a change.
You think you're doing great, if there is an outage in the one system you just get your important document from the other system.
Then one day someone accidentally copies and pastes the wrong data into the file. Now what do you do? If you goto your replicated copy it also has the bad data.
The answer is you should have had backups too. so you could go back an hour,a day or a week or even much longer. Depending on how much data you have, how often it changes, how important it is, and how quickly someone would notice bad data.