Finally I can uninstall `miniconda` and use the official python+pip.
I am no using python 3.6 (which comes with anaconda) create a venv with `python -m venv venv` and I couldn't be happier. Numpy and scipy wheels seem to work (although I needed an `--only-binary scipy` flag added, or something of the like).