Honest question: If a pixel is a point sample, why do all the down-sampling algorithms that look good involve some form of averaging the values of neighboring pixels? Wouldn't it be more accurate to pick one sample and throw away the rest?
But your intuition isn't wrong. Nearest neighbor can be a bit more "crisp" feeling on certain images, especially natural photos. A lot of photos have enough blurriness in the 1-3 pixels range that nearest neighbor sampling is a good choice if you're just resizing an image by a factor of 2.
But to see why nearest neighbor isn't the best in general, imagine down-sampling a checkboard and what happens as individual checkers become smaller than a pixel. You'll get moire patterns if you don't filter. You'd also expect pixels to turn grey as multiple checkers land there, and maybe not stick to only black or white.
Interestingly, some "sharpening" and up-sampling algorithms do the opposite - they subtract the value of surrounding pixels, rather than add. Those are assuming that some averaging already took place, and trying to un-average, if you will, to restore the original point sample.
The long answer is... really long.