Revisiting Daala Technology Demos

As it says, at this point it's getting pretty subjective, and so you'll need a blinded subjective test to really judge which one is better:

  The improvement in quality compared to the previous 
  status update is quite obvious. Daala is now much better 
  than both WebP and JPEG (libjpeg-turbo is the most 
  commonly used JPEG encoder). As for Daala vs BPG/HEVC, 
  the artifacts are obviously different and hard to judge 
  from just four images. Opinions are likely to vary based 
  on the viewer and the input image. At this point, what 
  we'd really need is a full subjective test. Fortunately, 
  this is exactly what is going to take place shortly, as 
  Daala has been submitted as a candidate for the Image 
  Compression Grand Challenge at ICIP 2016. The results 
  should be available in September. In the mean time, you 
  can read the paper we will be presenting.

What I generally do is toggle between three images, the original, Daala, and HEVC, and focus on a different area each time. There is always some obvious distortion from the original, so I try to determine for each area I'm focusing on, which one is closer to the original perceptually. It is a subjective process, but if you make this evaluation for each area of the image, count up your judgements over the whole image, do that over multiple images, and have multiple people all do that and combine their results, you can come up with a reasonable evaluation of the quality.

The visual quality is a bit of a red herring, as in most situations you'd just up the bitrate until it was acceptable. So quality and bitrate are in some senses interchangeable.

The graph above the comparison suggests that at this low bitrate, HEVC/BGP will very slightly outperform Daala on average, but each individual image may be slightly better or worse. It'll more probably come down to which kinds of artifact you're more familiar with and so see more easily.

Switching quickly between the original and the encoded version is probably the easiest way to spot the artifacts/compromises that are being made.

True, some ready-made video comparisons would be nice, but the codec is being actively developed as we speak, so they'd be out of date pretty quickly (the bitstream format isn't even frozen yet).

However, the Daala code is open-source, so people are free to generate their own comparisons using whichever codecs they like on whatever video snippets they have.

The Daala code is available via Git (https://git.xiph.org/?p=daala.git;a=summary and https://github.com/xiph/daala/) and there are even some test video sequences one can compare with at https://media.xiph.org/video/derf/.