I am trying to call attention to the models used for evaluation comparison. There are 3 factors: inference speed/latency, model size in total loaded VRAM, and model performance in terms of output.
Comparisons should address all of these considerations, otherwise it's easy to hide deficiencies.
https://cdn.prod.website-files.com/64f4e81394e25710d22d042e/...
What I want to see is comparisons to NON-quantized models all with around the same VRAM along with associated inference latencies.
Also, we would want to see the same quantizing schemes applied to other base models.. because perhaps the paper's proposed quantizing scheme only beats others using a particular base model.
They also show it has little to no effect relative to fp16 on these models.
IMO that's enough. Comparison against smaller models is much less useful because you can't use the same random seeds. So you end up with a very objective "this is worse" based purely on aesthetic preferences of one person vs another. You already see this with Flux Schnell vs. the larger Flux models.
They report that their method produces a model that is 6.5 GB from flux (22.7GB). Why wouldn't you want to know how their 6.5GB model compares to other 6.5GB models?
Regarding aesthetic prefs: it's an open problem what an appropriate metric is for GenAI... LLM arena is widely regarded as a good way to measure LLMs and that's user preferences.
In any case, the authors report LPIPs etc. They could do the same for other small models.
But you're quite persistent in that they need to address this, so it seems much more likely they silently added it after your original post, or you didn't click through, concern trolling would stay more vague
isnt that the first image in the diagram / the 22GB model that took 111 seconds?