For Science: Does ZFS deduplication work on intros of TV shows?
manuelgrabowski.de
manuelgrabowski.de
Many media formats (including all digital TV formats) encode a rolling hardware timestamp of 33 bits (or more) which won't be the same for two separate segments by random chance. Audio, video, subtitles and other metadata will be ordered differently in the stream because they all comes from sources that have their own separate clocks. Synchronization between clocks at different stages in the media pipeline will cause different frames in the sequence to be dropped, padded, made into keyframes, etc, which then affects every bit in subsequent frames. TV stations use time-based watermarks. TV stations use digital compositing software that may retain bits from previous frames on an ongoing basis. Many studio media pipelines still involve analog steps.
The list of complications goes on.
Unless your pipeline is lossless, uses only digital sources and uses totally synchronized clocks for all stages, you're about as likely to get an MD5 hash collision by accident as you are to get any two non-trivial sequences of compressed video to be bit-identical.
As for compression being largely ineffective, the shows are already compressed anyway. In fact file system compression seems to be less relevant these days as most modern file formats have compression built in (even Office documents are just ZIP files). But as the author said, the overhead for compression isn't damaging for the performance* of ZFS - unlike with deduplication
* ZFS compression is particularly performant† when using the newer lz4 algorithm available in OpenZFS - which I'd highly recommend people using if they're not already.
† I know "performant" isn't technically a word. But it should be.
My "performant" footnote was just an attempt to pre-empt such attacks; though ironically including the footnote has now sparked the tangent itself.
For instance, "performance" in terms of time is relevant to all IT systems, but there is usually some other dimension, for instance, compression ratio or accuracy, that matters too.
Thus, "fast" is a better adjective when it applies because it is more clear.
As for accuracy, well you'd expect any and all deflation compression to be 100% accurate anyway. Anything less than 100% would corrupt your data (unlike with audio / video "lossy" compression where you can remove / group non-perceivable data)
Alternately, you could decompress the intro frames - that might get ZFS dedup working, if the length of the file header is the same with all episodes.
Those groups often seem strangely bleeding-edge. They always adopt new features years before western groups will... I wonder why.
As a result they're always on the hunt for options which could help them do their stuff and:
* they'll find out about other possibly interesting features at the same time as they're poring over release notes
* because they might have to re-export their rips multiple times during QC, they can toy around extensively with features & settings (especially on early exports when they'll most likely have typos & al and will have to redo the export anyway)
tl;dr: they have plenty of opportunities to find and try out bleeding-edge features.
I think the long answer has something to do with what they do benefitting from it more than other things, and the short answer is "because they're geeks"; so they perhaps care less about slow hardware adoption of things like 10-bit than other encoders.
Changes to those standards can cause significant upheaval, as happened when the standard def TV groups multilaterally agreed to switch from XviD in AVI to h.264 in MP4 container.
AIUI, the tertiary anime groups, who take h.264/MKV/ASS releases and re-encode as MP4 with burned in subtitles, cater to those using lower spec hardware but who still want high def video.
You would then quickly discover that the chances of huge gains would be small, without ever even looking into the ZFS aspects.
Oracle seems to be doing alot of the lately where they are posting really old collaterial
Also, wouldn't proving it doesn't work by setting up a deliberately flawed scenario amount to a bit of blogsturbation? Should I write an article about "finding out whether my computer will turn on even if it's unplugged"? Answer: no. But read my article about me TRYING it just to be sure.
I think you should write a follow-up article where you splice together raw video files and include a similar segment in all the different files. Then, put that through the dedup test. I would actually be interested to see the ability of ZFS implementations to find worthwhile anchor points within files and do smart byte-level deduplication.
And especially for rips from an already-compressed signal source (like TV), the odds are well against a sequence, although identical visually, encoding into exactly the same sequence of bits digitally, as there are so many factors that can vary the datastream you receive, even before it's ripped and re-encoded.
To stand any chance of this working, you'd really need to be storing and comparing uncompressed frames direct from the master, before ANY kind of variable encoding or compression. But if you have enough storage to be working with files on that scale, deduplication is probably not your biggest concern. :)
The seeking might make it impractical.
Hence any compression applied will always produce different results between episodes of a show. This would make any de-duplication extremely difficult.
There will always be some sort of noise, pixels aligned differently etc., in a production like a tv series, and expecting the encoded output to be identical/matching on a blocklevel is pretty naive to say the least.
I did some experimenting using ZFS deduplication on MPEG2 files, where I encoded hundreds of MPEG2 dvd-sized videos, where 90% of the material was identical (the last 10% was affected by applying different watermarking techniques to the footage), and got some decent deduplication ratio (x1.2:1 or so).. But ZFS deduplication is expensive in memory/SSD, and it was definitely not worth it.
I'm writing some technical documentation for a project right now - perhaps I'll put in a few tasteful images from Baywatch to make sure the project managers who have to read it will make it to the end.