> A lot of open sourced CS research is not reproducible.

I'm not sure if this was meant to be a counter-argument to me, but I completely agree!

> "the code still runs and gives the same output" is not the same as reproducibility.

Yes, bit-for-bit identical results are neither necessary nor sufficient for reproducibility in the usual scientific sense.