Not only do you upscale with nearest neighbor then downscale with bilinear, you also make sure that you are using the correct colorspace for downscaling.
The most common mistake is to reinterpret sRGB as if it is a linear color space, and assume that half of RGB 255,255,255 is RGB 128,128,128. But that's wrong. sRGB is a gamma-compressed color space that does not respond linearly.
First you treat all sRGB values as if they are between 0 and 1. To convert into linear RGB, take all values to the power of 2.2 (like squaring them). Then you do any math on color values in Linear RGB. For example, half of linear 1 is indeed 0.5. To convert back into sRGB, take all values to the power of 1/2.2 (like a square root).'
This means that 0.5 in linear RGB corresponds to 0.73 in sRGB. You can confirm this by placing a black and white checkerboard next to a square of RGB 186,186,186. They will match in apparent intensity (provided you have an sRGB display that doesn't have viewing angle problems).
Some graphics APIs can do the sRGB to Linear RGB conversion for you automatically, but you need to request the API to do that. This makes the pixel shader only see Linear RGB, but the pixels are coming from an sRGB texture, and are being written to an sRGB buffer as output.