conda doesn't just package python libraries, but also the C/Fortran/other bits that the scipy stack often depended on. With the rise of binary wheels that is less needed though