For instance you should be able to install a numpy that is compiled to take advantage of your specific CPU. You should be able to pick at install time which one you want WITHOUT patching the the source code directory in any way that would make it back to version control.
I suspect it's the features of the BLAS that will matter, though, a lot more than Numpy itself. The Numpy build system by default prefers MKL, OpenBLAS, then ATLAS, and I don't remember what after that, and it's up to you to compile those with the features you need.
I didn’t feel in control of Python builds until I had a wheels only policy. (For one things you can download about 2% of a wheel and get the dependencies to put into an SMT solver and globally solve the dependencies before you build anything. pip, poetry and similar systems use eggs and have to run setup.py to figure out deps which means they solver locally and eagerly and back themselves into corners that they may or may not be able to get out of.)
If I have to build from source I want to build a wheel, archive it, then reference the wheel in my repository. I don’t quite have the tooling to make it as easy as it really is.
In python I would create different requirements files together with freeze versions of those. One could contain the standard "numpy" and another would be "numpy @ my-custom-built-numpy-url". I haven't tried it and don't know how it would work in practice.
This doesn't allow you to use an alternative index, but it does allow you to override packages that restrict their dependencies versions too strictly