MS Python: How to deal with the pain of “unable to find vcvarsall.bat” | Hacker News Reader

MS Python: How to deal with the pain of “unable to find vcvarsall.bat”

blogs.msdn.microsoft.com

45 pointsbrettcannon23 comments

This is a very odd blog post in my opinion coming from MS. It acknowledges the pain of dealing with source distributions on Windows and gives excellent advice on how to overcome the issues. However I think it really misses the forest through the trees.

If MS was really serious about Python being a first class citizen on Windows then fix the root problem which the post acknowledges, ".... Because Windows has a different culture, most people do not have (or need) a compiler." Why doesn't Visual C++ express come installed or as a super easy to install dependency on Windows? Howabout MS pushing to change the culture of Windows to one where compiling from source isn't a big deal and is as easy as it is on Linux?

Also it's super helpful that this post lays out where to find precompiled versions of Python packages for Windows, but again it's something if MS were really serious about they would step up their game significantly. Why is it that some random person, Christoph Gohlke (and as a side note many thanks to them, they have undoubtedly helped thousands of people), is the maintainer of one of the most important resources for Python users on Windows? How about MS steps up and dedicates resources to ensuring precompiled and tested python binary packages are available for all packages?

Finally I think the message that people should bug Python library owners to get windows wheels up is very poor advice. Library maintainers have enough to deal with and piling on more frivolous issues won't help. What if the maintainers don't even have Windows machines to build binary packages with (and again, why doesn't MS step up to provide them with said Windows machines/licenses)? Badgering the maintainers isn't going to make things better. If MS is serious about python on Windows they need to step up their support and make the platform better.

  Why doesn't Visual C++ express come installed or as a
  super easy to install dependency on Windows?

In theory that's what VCForPython27.msi (and the others) are supposed to be. And in theory it isn't too difficult to download and install the Express or Community editions of Visual Studio 2015.

A Windows feature that installed the matching C++ compiler plus platform SDK would be freaking AWESOME.

  as a side note many thanks to them

No kidding. Us Python on Windows users really owe them a debt of gratitude.

Last time I tried doing this, it was very unclear to me if there was a way to download the VS compiler without giving an email address or logging in or something to Microsoft. In the end, this was one of the big drivers that pushed me over the edge to switching to 100% Linux.

This is the link where you download the MSVC for Python 2.7 installer: https://www.microsoft.com/en-us/download/details.aspx?id=442...

Once it loads, close the survey popup thay may or may not appear and hit the download button. No need to login or put your email or anything.

I thought you were going to say that even with it installed, there are rare ocassions where the system doesn't acknowledge the existance of MSVC, for example Cython.

I thought the post might give a solution to that particular issue, but so far it seems nothing has changed in that aspect.

Well, I don't need it now because I'm off Windows permanently, and I would use Python 3.x even if I were, but it's good to know that something like this is available. I was sorta wondering if the situation had changed since I looked about 18 months ago.

It's not first class support but it goes a long to patch potholes in a bumpy road to installing extensions from source. Motivated but unkowledgeable users are much more likely to succeed with this info while the benefit between just a blog post and having it built in to OS may be negligible (while the technical investment of building the compiler into the OS could be significant).

By the way MS provides free official VMs for testing IE but work just fine for building wheels.

Yeah I've used those VMs and it's nice but they're only valid for 90 days and IMHO the licensing is kind of murky (it says they're only for testing IE). You also don't have a choice of 32bit vs. 64bit VM (I've noticed the win 7 VM is 32 bit, the win 10 VM is 64 bit for example). There really should just be 100% free for any use VMs that never expire for library maintainers to use.

On the other hand, strawberry perl (http://strawberryperl.com/) for windows solved this by simply including a free toolchain (gcc). The gcc compiler for windows is hardly typical, but it was an interesting choice so that they could have more control over the build environment.

That said, I think MS can/should make installing a toolchain easier than they do. It ought to just be another component of the OS that you can choose to have or not have. Supporting different versions of the compiler on one box is a more advanced use-case that advanced developers can deal with - but each rev of the OS should have the "default" toolchain that installs seamlessly.

  If none of these options is available, you will need
  to consider building the extension yourself.  In many
  cases this is not difficult...

This is horseshit.

I spent a few weeks at the beginning of the year building MSIs for SaltStack and its dependencies, as I wanted a fully native way to bootstrap a Windows client installation via GPO using a native Python installation instead of SaltStack's NSIS executable installer, which uses an esky build. In short:

* The compiler included in Microsoft Visual C++ Compiler for Python 2.7 is not compatible with distutils.

* The compiler included in version 7.0 of the Microsoft Windows SDK is not compatible with distutils.

* The compiler included in version 7.1 of the Windows SDK is not compatible with Python, nor is it compatible with distutils.

The only combination of compiler plus SDK that I could get to work properly was Visual Studio 2008 and Windows SDK 7.1, and that still didn't work quite right due to bugs in the code underlying bdist_msi, source distributions not including everything used to actually build a given package, incorrect package build instructions, and so forth. (For example, I seriously doubt that the pywin32 developers even follow their own package build instructions.)

In case you're curious:

https://gist.github.com/xenophonf/c2351f8af80d3231b547

Ultimately, I've given up and switched back to SaltStack's executable installer, run out of a computer startup script (a page out of my own book on 90s-era NetWare sysadmin, yuck).

Not trying to be rude, but that seems more like an issue with lack of research, for all of your initial troubles.

Python was compiled by Visual Studio 2008, up until Python 3.5, and I believe that was made clear in the extension module documentation for python.

Having used 3.5 and VS2015 (after waiting for multiple days just for VS to finish installing), everything else has worked pretty spiffy since

You're not being rude at all.

The extension module documentation, including this blog post, says to use Microsoft Visual C++ Compiler for Python 2.7. I couldn't get it to work properly, and I tried the other toolchains after quite a bit of research, a word which here means "reading through the Python source code and trying various search terms in a vain attempt to solve a problem that apparently isn't a problem for anyone else, but they can't be bothered to specify their obviously functional build environments in sufficient detail for me, an expert in the art, to recreate in a clean virtual machine here at home". It was only after this little endeavour of mine that I came to appreciate people's work on reproducible builds, Vagrant, etc.

And you aren't kidding about that VS2015 install taking forever and a day. Updates are almost as bad.

I guess I knew in general about wheels but didn't see how it would solve "usability" problems like this one.

Does `pip install lxml` fail in a similarly perplexing way if you don't have `build-essential` (`gcc` et al) installed? If so maybe the difference is that the posix/linux crowd is more likely to have installed a compiler at this point.

Here's hoping MSFT's Linux ABI support evolves into the norm for a port target.

  compilation terminated.
  
  *********************************************************************************
  
  Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
  
  *********************************************************************************
  
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Cleaning up...
  Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_test/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-QEyvN6-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_test/lxml
  Storing debug log for failure in /home/test/.pip/pip.log

There is some stuff above in the log, but the nothing about build-essentials. And a new dev would likely have to hit google/stackoverflow.

Was that error because gcc was missing or because libxml2-dev wasn't installed?

Touche, python-pip on ubuntu has a dev requirement on build-essential. So it should be rare that it's not installed.

I tried to get into linux development within the last year or so. One of the most frustrating obstacles I ran into was that attempting to install packages would mysteriously fail. I found it that was because I was missing a package called "build-essential". I thought one of the package manager's reason for existence was that it could track dependencies, and it didn't appear to be doing that. I still don't understand how that can happen. It seems to me that a package installation ought to be tested against a fresh image to verify the dependency tracking is working correctly.

Packaging dependencies and build dependencies are tracked separately. For example, if I don't have a C compiler installed, installing a development package like "libxml2-devel" will probably cause the compiler to be installed.

In the opposite case, say that I have the compiler installed, but not the dev package for the library. Compiling an object will fail, if it #includes a required header, but that will be in the form of a "file not found" error emitted by the compiler. Similarly, if I tried to link a program and include the "-lxml2" flag, the linker will reply "cannot find -lxml2", meaning that libxml2 isn't installed, so it can't be linked against.

Generally, we'd say that it's not the compiler or linker's job to talk to the package manager. The build configuration system included with the project (autoconf scripts, cmake or ant profiles, etc) ought to check for prerequisites.

build-essentials is an Ubuntu package which depends on a number of important libraries, gcc, the associated toolchain, etc. Just installing a pre-compiled package won't require a compiler though, it's only when you want to build something from source that you need it. There are a number of python packages that don't come pre-compiled; that may be what you were seeing.

> If so maybe the difference is that the posix/linux crowd is more likely to have installed a compiler at this point.

The long running issue with Windows was that there weren't any compilers available (AFAIK) outside of installing Visual Studio first. When Visual Studio started offering free versions, the problem persisted: it's rather hard to justify downloading and installing a +4gb program just to make an dependency or extension to work (even if it wasn't directly required for the said dep).

For example, this was a huge issue with node-gyp[1], which prompted Microsoft to release standalone Build Tools[2] to alleviate the problem. I think it's the same kind of issue with Python here, right?

[1] https://github.com/nodejs/node-gyp/issues/629 [2] https://www.microsoft.com/en-us/download/details.aspx?id=499...

`pip install lxml` will indeed fail without VC++ for Python. In fact, sometimes it fails even with VC++ (e.g. installing lxml from within PyCharm). However, executing the binary installer or using `pip install lxml_package_whatever.whl` works just fine.

Probably one of the most difficult errors as a complete noob developer to hurdle.

It's also part of the reason why I ultimately switched to Mac.

nice! wanted to install mysql-python and was stuck here!! Thanks.

Use pymysql instead. It works now, and it's 100% Python.

Reply on news.ycombinator.com