In test of objective metrics SSIM was one of the best (0.8 spearman correlation with human scores) http://www.ponomarenko.info/tid2008.htm
IHMO the study has been done very well. They've even eliminated potential differences coming from colorspace conversion and chroma subsampling. If anything, this study favors WebP that can't do full-resolution color.
However it still doesn't accurately represent preceived quality; time and time again certain psy optimizations for instance are shown to reduce SSIM while perceived quality goes up. Conversely, a codec can optimize for SSIM alone and score well on a test that ranks by SSIM; but the perceived quality could be better. For this reason e.g. x264 has an option to disable psy optimizations that hurt the objective metrics.
There are other things that can hurt metrics a lot while doing nothing perceptible, though these are rather like bugs or errors in testing methodology, and as such would likely be noticed in a well conducted study. See also: https://wiki.xiph.org/Notes_on_testing_theora#General_points...
BTW HEVC-MSP is limited to 4:2:0 only as well.
If you use a greyscale/luma-only metric, you have to encode greyscale from the start, not convert a color image after encoding. Otherwise you're measuring chroma bit cost, but not chroma distortion. Thus, invalid.
Actually, is PSNR-HVS-M greyscale-only as well? In that case, only the RGB-SSIM results would be valid...
And actually, thinking about it even more, converting to greyscale at the tail end just for Y-SSIM and IW-SSIM is fundamentally flawed (if it doesn't result in exactly the same bits as the luma plane), as now you're measuring distortion from an image that the codec was never asked to encode!