That would allow the subject to easily tell the difference between the two samples by looking to see which had all 0's in the 4 lowest bits.
Are you saying the "16-bit" sample files actually had noise added to the least significant 4 (8?) bits? This is not dithering. Dithering is adding noise BEFORE truncation. A dithered 16 bit rendering of 24 bit audio will only be 16 bit. An undithered 16 bit rendering of 24 bit audio will also only be 16 bit.