Welcome to the Python Engineering blog | Hacker News Reader

Welcome to the Python Engineering blog

blogs.msdn.microsoft.com

147 pointstanlermin61 comments

Side note about the author: Steve Dower from MS has also been helping out core Python development. He's done great work on improving how Python gets built, maintained, packaged etc on Windows.

Although I'm not a Windows user, it has been great to see the progress made in that area recently.

He's also a nice fellow and pretty good at pool.

Hi, team lead here. Very Happy to see this on HN.

If there are any requests for the team pls feel free to voice them. Smiles, frowns, ...

Anything from making pkg installation easier to cross platform dev tools.

Cheers.

> making pkg installation easier

http://www.pyinstaller.org

Best thing since bread. OK, sliced bread, but it's close ...

Thanks ableal - improving pkg installation is under heavy discussion currently. We'll take a look. Cheers.

Actually, I think support for nuitka would be amazing.

Its already 2x faster than cpython on pystone, but its even more promising to compile everything into a binary blob for easy distribution.

http://nuitka.net/

Thanks tanlermin. We are aware of nuitka and in general very excited about the various Turbo options that are showing up for Python.

Our own effort is pyjion and we have a talk at PyCon 2016:

https://github.com/Microsoft/Pyjion

The high order bit is to restructure CPython to enable easy plugging in of various JIT optimizer/codegenerators into the interpreter. Our own attempt will be based on the cross-platform .Net JIT (generating machine code).

This is really cool..hope to see it succeed :)

I make a Python extension for SQLite (APSW) and by far the single biggest Microsoft annoyance is the compilers. I support Python 2.3 to 3.5, and see downloads of the binaries for all of those versions (!), both 32 and 64 bit where relevant.

Getting a system setup where each Microsoft Visual Studio version is present, and then having to use the Resource Kits for 64 bit compilers is very painful, and likely impossible to repeat (the various installers randomly disappear). Should I ever lose the system, I'll probably give up on providing Windows binaries.

Microsoft do already provide a small compiler bundle for the most recent version, but that doesn't reflect the user base. I would be delighted if Microsoft provided a more complete compiler bundle (no guis, only enough to keep the various versions of Python happy). Or alternatively do what the IE team did and provide virtual machines preconfigured. That would make supporting Windows a far smaller burden.

roger - 1st a big thank you for the wonderful service you provide.

I assume you are aware of this:

https://blogs.msdn.microsoft.com/vcblog/2015/11/02/announcin...

The python team actually pushed for this unbundling. Which other versions do you need? I can take than info back to the VC++ team.

There's also this for VM's:

https://blogs.msdn.microsoft.com/visualstudio/2015/01/08/azu...

Thanks.

Those are the "most recent version" I mentioned. The Visual Studio versions and Python adoption are out of sync. To build for the versions of Python I do, VS 2008, 2010, and whatever Python 3.5 needs. Note that the free versions excluded 64 bit compilation so the resource kit had to be used.

Here is my setup doc https://github.com/rogerbinns/apsw/wiki/Setting-up-a-Windows...

Thanks rogerbinns - Understood, just wanted to make sure you were aware & it looks like you are :). I've pinged the VC++ team and will report back.

BTW we've setup the alias pythonbuildtools at microsoft.com where folks like you that are building pkgs can speak to the Python+VC++ team members. I recommended send your request there as well.

Cheers.

What happened to the MS-Django MSSQL collaboration? It seems to have fizzled out. https://groups.google.com/forum/m/#!topic/django-developers/...

Thanks for the heads up. It turns out that we are now much closer to the SQL team, so I'll ping the owner listed on that page & get an update.

Hello phonon, I work in the MSSQL team and have been actively involved in the collaboration. We have been in touch with Michael and plan on improving the landscape here. In the meanwhile, we would love to help you with any technical difficulties you are encountering. Feel free to email me at meetb at Microsoft dot com.

We also worked on a few samples that you can check out and give feedback: https://github.com/Azure/azure-sql-database-samples/tree/mas...

Some time ago I wanted to make a cross-platform app in python, I wanted to use the ui bindings of each platform, I had some success with Cocoa in OSX but when I tried in windows, I saw with IronPython I could use WPF but when I tried to use it, it was reaally outdated and there wasn't even IronPython for python3... I guess it is no longer in active development... In the end I used swift in osx and c# windows, but I would have loved to use .net apis in python and have the same codebase for all the platforms.

IronPython has been in the community's hands for a while. We do support it in PTVS, but that's about it. Curious - did you consider Qt?

I was trying to port an app I already had in c# with wpf to be cross platform, so some networking code could be shared, but I could still use xaml and all the wpf work that I already had (although it wasn't that extensive, just tried for the sake of learning), I also used some audio apis, and with IronPython i could call c# code and use NAudio, so in the end I couldn't do what I wanted with just CPython

I recently started doing more development on Windows and coming from os x and linux it's a bit weird to use `py` instead of `python` but other than that it's pretty straight forward.

I also want to mention that I'm loving all the recent attention for python on HN. It's a great language, 2 or 3, and has many great use cases and has a great ecosystem.

I used to use a Windows box at work, and still use one at home for games. Python is getting much better for Windows. But some of the package management still completely sucks. Cmndr helps out a lot, but some of the math heavy packages (I'm looking at you, cryptography) have to be compiled with the version of the Windows C compiler that was originally used to write the package, which is not always easy to find. Or you have to find a precompiled version, which is tricky to install into a virtual environment.

Of course, for general computing, things are good and getting better, and I really appreciate the work Microsoft has put into Python Tools for Visual Studio.

I love my sublime text, but I also have a mad crush on visual studio from my C# days. If VS would support Pyramid as well as it does Flask and Django, I would seriously consider switching to a Windows machine full time.

The compiler issue has been solved by Microsoft with MSVC for Python[0].

For anything that still fails with that (may happen) or if you need NumPy with Intel's MKL, check Christoph Gohlke's Python Module repository[1].

[0] https://www.microsoft.com/en-us/download/details.aspx?id=442...

[1] www.lfd.uci.edu/~gohlke/pythonlibs/

My biggest gripe with Python on Windows is all the libraries that sport a GCC toolchain for *nix environments, so completely fail to work. Yes, that is the library contributors fault, but I hate that Python even allows that.

Had to chuckle at the python on HN comment; Python tends to get universal praise around these parts (supposedly some investors won't even waste their time with you unless you're a Python shop).

If there was ever a race between Python and Ruby, then Python has won. Everywhere from Google, Microsoft, Facebook are now using Python.

This isn't to say Ruby is dead, but it will remains in a niche.

> Python is obviously a big part of data science these days

Off topic, but why is that? I mean, Python doesn't seem any more suitable to "data science" than Ruby or Lua or Clojure or any number of other similar languages. So why do I keep hearing that Python is super popular with scientists?

Because of the community and libraries; not the language.

Agreed. Plus Python being taught at more universities now probably helping.

Are there specific data science libraries or are we talking about the large number of libraries in general?

NumPy and SciPy are the ones that I know of (and I don't even write Python, so there is surely more).

numpy, scipy, scikit-learn, matplotlib, pandas, biopython, sage, ipython and ipython notebook — now jupyter — (with both matplotlib and clusters management integrations), non-C accelerators (numba, cython), anaconda (a science and engineering semi-proprietary bundling & package manager).

Many of these have their roots in the late 90s or early 00s. I guess Python got its fangs (haha) into scientific computing at the turn of the century, from the late 90s you find articles mentioning the introduction of scripting languages as glue in scientific pipelines variously mentioning Perl, Python and Tcl, and by and large it seems to have slowly accreted around Python.

Some of the Numpy and Scipy routines go back earlier than the 90's. AFAIK, they trace their roots to numeric routines on Fortran punch cards.

> AFAIK, they trace their roots to numeric routines on Fortran punch cards.

Oh yes they can bind to much older libraries (e.g. BLAS), I was talking about the packages themselves. For instance Numpy is the unification of Numeric and Numarray, the former having gotten started circa 1995[0] on matrix-sig[1] and the latter originally indented to replace the former. So numpy itself has its roots in 1995.

[0] https://mail.python.org/pipermail/matrix-sig/1995-August/000...

[1] https://mail.python.org/pipermail/matrix-sig/

There are gems in these early messages too, for instance one of the early threads suggested allowing parens-less tuples and slice literals (a:b) in tuples within indexing: https://mail.python.org/pipermail/matrix-sig/1995-September/...

Slice literals, let alone slice literals in tuples, remain indexing-only features to this day.

So the language has a long history of adapting itself to numerical computing concerns.

Why can't it be both? If the language is easy to adapt to then that's a plus to anyone.

The language probably had something to do with it, at least in that scientists got involved very early on[0] and language features were added as a result.

[0] https://mail.python.org/pipermail/matrix-sig/ note the direct involvement of GvR

Network effect and a few really big names in it?

And it is easy to use for people that want to get work done. i.e. clear syntax, and a goal of one-way to do things.

It's the libraries like NumPy, Pandas, SciPy, Matplotlib, SymPy, and so on that make the difference. There's been a ton of work put into developing those Python frameworks for "data science" which don't exist to such an extent in languages like Lua.

Lua has Torch.

Ok thanks for pointing that out, I stand corrected. Regardless I think Python has reached a sort of critical mass of data science packages.

Numpy existing (more than) a decade ago helped with getting mind share.

OK, now compare that to "literally everything else" in python.

A lot of those actually started as wrappers to existing C/C++/other libs, something that is very easy to achieve in Python and not so much with other runtimes.

To be sure, NumPy is the third implementation (Numeric, numarray, NumPy) of the basic array data type, so getting it right isn't as easy to achieve even in Python.

And when library X and unrelated library Y are wrapped, they often both use Numpy arrays, making them interoperable.

E.g. if you do geographical work with raster data, there is GDAL, a raster library with Python wrappers. Because it exposes the raster data as Numpy array, you can easily make images from them to overlay on maps with matplotlib, or analyze them using scipy. In C/C++ it would probably be a lot more hassle to combine them like that.

I don't think the libraries are the only reason python has a strong following -- after all, perl also has a huge collection of libraries. I realize there's a healthy dollop of subjectivity in all this, but here are what I think the reasons for Pythons success.

1. Python uses words where many other languages use symbols. In addition, the words Python uses are simple and clear. Instead of '!', '&&' and '||', python has 'not', 'and' and 'or'. Often constructs that look like function calls in other languages look more similar to english in python. For example: if 'Python' in names: print("Found it!") .

2. Python uses indentation to control scoping. The merits of whitespace sensitivity are of course open for debate, however, I think it pushes the code closer to what pseudocode looks like, and this is probably a good thing.

3. Python has a simple and coherent object model. Unlike Lua or Javascript, python has a real object system (i.e. not prototype-based) with inheritance. Although object oriented programming doesn't get a lot of love on HN, it does have several benefits, compared to, for example, functional programming. One advantage is that it arguably well-understood and knowledge about OO has already been widely disseminated. Another advantage of OO is that the transition of using structs to hold mere data (a la C or Pascal) to OO is a simple matter of adding function to the struct. With OO, you can start with code that is more procedural, and work your way towards a design that is more OO. This is a path that makes sense in the context of scientific programming, since one often starts with a short programming that, say, implements an algorithm in a single function. (Then you build on it in an OO way by allowing loading input data, etc.)

5. Numerous well-defined, easily understood, and easily accessible customization points. For example, in Python, allowing the user to write "x + y" is as simple as implementing the "__add__" magic function for the class in question. Ability to overload the mathematical operators is critical for a language used in a scientific context, and this for example essentially eliminates Javascript. It also eliminates lisp in all its variations due to its lack of infix syntax (short of writing custom reader macros). The story is similar with decorators.

6. Python is easy to extend and embed. For some reason, everyone seems to rave on about how great Lua's C API is. Frankly, having written C++ extensions for both Lua, Python, and Java, the one I found easiest was Python's (especially -- but not necessarily -- using boost.python), followed by Java (especially using JNA), and Lua I found quite painful due to the explicit manipulation of the VM's stack.

7. Others have mentioned it, but the libraries are obviously a huge part of Python's success. As others have talked about it, I won't say anything more about it here.

6 is a very good point - Lua has advantages for embedding, but its API is not one of its better points.

Of course, when coding in C, your options are limited - Python's reference-counting isn't much fun either! But code using the Python API tends to be relatively readable, even though you run the risk of forgetting to put a decref in. Lua API code on the other hand, for all that it doesn't need anything like Python's incref/decref, has a tendency to be rather inscrutable.

but why were these written in python as opposed to ruby.

Timing and focus.

People in scientific computing started using Python the mid-1990s. This included Numeric (ancestor to NumPy) which Jim Fulton, Jim Hugunin and others started in 1995. Ruby 0.95 wasn't released until the end of that year. Moreover, van Rossum tweaked Python so it would be a better fit for matrix computing, such as multi-dimensional slices.

Then there was PyFort by Paul Dubois at Lawrence Livermore National Lab (1999), and SWIG by Dave Beazley at Los Alamos National Lab, which had Python support by 1998 (see https://web.archive.org/web/19981212033200/http://www.swig.o... ). Those made it much easier to access existing scientific libraries through Python modules. (A phrase at the time was that Python would 'steer' the low-level high performance code.)

While at this time, Ruby was just becoming known in the English speaking world, and didn't really hit the mainstream until Ruby on Rails in 2005. This means Python had a 5-10 year head start, and Ruby hasn't caught up.

http://programmers.stackexchange.com/questions/138643/why-is... covers a reasonable and diverse set of explanations.

From personal experience, having programmed in a small number of languages C/C++, Java, COBOL, Pascal, Python, Ruby, various shells, Assembly (Intel) etc, there are a few things that keep attracting me back to Python:

a) Somehow the fact that indentation is meaningful forces my code to be neat automatically without having to run any beautifier through it. I consider myself a bit OCD on how the code looks (can't explain it) and I like how Python code finally looks. I like code to be maintainable and self-explanatory without leaving puzzles for my successors to solve.

b) Batteries included. Granted there are libraries for most things in other languages, but nothing with the ease of access as Python.

c) Portability without having to know the compiler intricacies of each Port/Platform/OS

d) Being able to deliver solutions on time and within budget without putzing around with irrelevant stuff that the customer never cares about.

Not being a Data Scientist, I don't know if any of this is relevant. But being educated in CS and responsible for delivering on promises I care about some of this a lot.

a) Having "beautiful code" helps keep focus where it matters: the "difficult" sutff (be data data science, machine learning, math problems, physics simulations, etc).

b) True, there are beautiful libraries all around, and easy to use. In particular to data science, I think NumPy+SciPy+matplotlib are comparable to MATLAB. With beautiful code, portability and no need to pay an expensive license.

c) Totally agree. Though there are small nitpicks in different platforms, most of the times they can be overcome.

d) Totally agree.

Also: the Anaconda distribution solves most of the problems that anyone could have with (b), (c) and (d).

I heard from a couple people that they don't exactly want to learn Fortran. Python stays out of your way and there is a major ecosystem around it by this point.

I saw they ran a small "python for data science" intro course this year on edx, so I figure they're definitely behind using the language in some capacity.

https://www.edx.org/course/introduction-python-data-science-...

I checked out that course a little; it looked very basic, at least the part I checked. Then left it.

Saw the course via an ad, FWIW.

AFAIK, Ruby doesn't have anything like Python's numpy. There has been some effort to fix that, but not one driven by necessity or an organized movement: https://github.com/SciRuby/sciruby/wiki/The-SciRuby-Manifest...

Fun bit of trivia: Hadley Wickham (of R's ggplot2 and dplyr fame, among many other things) said that if R hadn't been around when he first got into statistical programming, Ruby would have been his next choice, because of Ruby on Rails. He says now, his choice would be Python or Julia ("or maybe JavaScript"):

https://www.reddit.com/r/dataisbeautiful/comments/3mp9r7/im_...

Fun to imagine ggplot2 in R...or as a competitor to D3 in data visualization.

As well as the packages that people have mentioned, there are nice bundles like Anaconda or Canopy which makes it easier to get started.

Certainly working with people who know data but don't necessarily know much Python (even if they know it they might not know tools like pip, or want to spend time understanding the versions of their tools) it's way easier to say "just install anaconda" than "install python and all these packages we use".

I don't know if other languages have these bundles but Python's are very good.

Additionally, the jupyter/ipython notebook environment is very good, but I'm not sure that's converting people, I still see many researchers working in plain text editors.

It's in a sweet spot. It's good enough for unstructured exploratory analysis, for production code, it's super easy to get started in, and there are some good libraries. There's really low overhead to poking around data in it, and you can put something into production with it.

They are all fine languages. There is another (key) reason why Python is keeping its momentum: the availability of skilled, professional python developers with a lot commercial experience.

I've been called in a number of times to help out small businesses or teams that've started using Python because they were: researchers, scientists, or even just regular developers, and have found themselves with a growing business and customer requirements. Having a large, professional body of capable of (especially contract, here in London) developers makes a world of difference. You can find expert Julia, Haskell, Clojure, etc. developers -- but they are fewer and far between.

I'm not a scientist, but I think the answer would be orthogonal to why I (and probably many others) pick Python.

Is it tied to a single source?

No, many implementations exist and with the 2/3 split, the BDFL is weaker than ever (a good thing, overthrow all BDFLs).

Is it portable across platforms?

Yes. It runs anywhere C89 will.

Does it have a broad range of uses?

Yes. Already in largescale use for systems, web, administration scripting, game scripting, 3d applications, data modeling, game creation, crossplatform GUI applications including iOS/Android.

Is it easily maintainable?

Yes, as much as anything else. Nothing is a cure for the complexity issues at 500KLOC scale, not static typing, nothing.

Will it be around for a long time?

Since 1991, so likely in some form yes. The syntax has proven too popular with folks already. I'm considering creating own Pythonic language.

Is it popular, to get help easily and borrow code?

Yes. #3 on Github (practical comparison) and #5 on Tiobe (abstract comparison).

Since the data science folks don't also have a goal of becoming programmers, a language that had some early design goals of being easy to teach (see also CP4E [0]) and has appropriate "good enough" libraries lets them build custom analysis software without having to spend nearly as much time learning programming instead of more relevant things.

Also, inertia:

  * Python, 1991
  * Lua, 1993
  * Ruby, 1995 (Japan) 1999 (English)
  * Clojure, 2009

[0] https://www.python.org/doc/essays/cp4e/

The early design goal of being easy to teach also caused python to have a culture that places a high value on doing the work to make the developer experience better for beginners. You can see this in things like the Django tutorial, but it is more powerfully felt as an undercurrent in conversations at python conferences (based on PyTennessee, PyConUK, DjangoCon, and PyCaribbean)

Not a windows user but for what is written in the post, it looks like it will cover more transversal topics. Will follow it for a while, to see what it brings to the table.

Reply on news.ycombinator.com