PyCon 2015: Are we still changing the world?
blog.lisnr.com
blog.lisnr.com
The focus is on getting things done, today and tomorrow; not be trendy or be on the first page of HN or disrupt the language space using a new paradigm.
Python is "silently?" used A LOT. Not only on web development (which has more the spotlight on the Internet) but in areas like systems scripting, data analysis or science research. I think that is a great language with a lot of opportunities (including career opportunities) that is still growing.
Yes, it's not the new kid in town. I think of it more as an advantage, really.
I agree about our significant use outside of web dev -- even though I know a lot of it is "out there", I had a hard time gathering it with many major orgs not being on CrunchBase, AngelList, or StackShare, (and some with a footprint that doesn't represent their true size on GitHub). If there's a source for this data out there, I'd love to find it.
But there's lots of other things that can change the world besides programming languages. Are programming languages really the best place to hang your hat? Do you want to compare Python to Ruby or Node.js, or compare Python a web application or wiki or the cloud; i.e., entirely different categories?
0: http://pyfound.blogspot.com/2015/02/psf-python-brochure-sold...
It's the things (end product) that matter, and unless they could only/better/easier be implemented in $LANGUAGE, then it could have been many other languages that could have been used as well.
Whereas something like C did change the world.
While I think, say, other areas also get things done, Rails developers are more likely to blog and have pretty fonts. Nothing against that, it just seems to be the case.
While metrics show there's more Python out there, it's just not exciting, it doesn't change a lot, and it just works. And that's fine too.
With respect to metrics, I think having the baseline is important, and you're right. I also found myself looking for more granular metrics then what I was able to find. I really want to be able to answer questions like
- "Within Python, how is usage shifting between web apps, scientific work (both research prototyping and in practice), open source, machine learning, etc.?" - "How are the 'frameworks on top of frameworks' like Flask-RESTful and Django Rest Framework changing in adoption over time?" - "Is anyone actually adopting Python 3 as their default yet?"
I wasn't quite able to find that data, but if anyone has ideas for first steps in getting there, I'd be happy to contribute (email in profile).
Python was at least previously big in a lot of local shops doing a lot of systems programming type stuff (management apps) -- including folks like Red Hat and HP, including my old one (Ansible). Startups are still highly likely to use it.
There are enough Django shops around, including at least one major Django consultancy-type shop, but it feels like slightly more (but not much more) rails on the web side.
I'm not plugged into the industry biotech to know how much Python is floating around, but I think it's probably pretty common, just that those folks travel in slightly different circles and we don't cross polinate enough.
"Is anyone actually adopting Python 3 as their default yet?"
Almost certaintly someone has, the real question is what percentage and what characterizes them :)
I stopped using Python for data science and use R and it really worked out great for me. R has gained a lot of momentum and I think that Pandas with Wes leaving isn't being developed as quickly. (Most of it is re-doing what R has done). I am amazed at all the FUD I was told to not use R and just use Python but after hitting so many walls I tired it and I am so glad I did. Your millage may vary.
My main beef about R (and similar tools) is that I summarize my data I get from Hadoop and find it is still too big for R to handle and I then have to summarize it some more.
[1] https://blog.javascripting.com/2014/07/28/fastest-growing-ne...
[2] http://www.pcworld.com/article/2480920/r-programming-languag...
[3] http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15...
Yes I have read [R Inferno] http://www.burns-stat.com/pages/Tutor/R_inferno.pdf
By the way i should have stated that Python is a good choice for doing Data Science it is just that R works better for me.
You assessment is pretty accurate. R is almost the opposite of Python -- you HAVE to use lots of external libraries. Whereas in Python, the core is super solid, but the libraries you get are of varying quality.
R took me the longest to learn out of any language. I currently have coworkers struggling with R, and cursing it. But the learning pays off... it's pretty much the only option out there right now (and yes I use Python for almost everything, but you need R too.)
What walls and why did you switch? Did you find the shift in syntax and mindset difficult? Have you used rpy2?
There definitely seems to be more innovation in R world, but blaze, numba and bokeh have no R equal.
I work at a Microsoft Office Universe (They refuse to know of any tools besides Word and you are a tech god if you use Excel) So when I finish my reports in R I actually give them Office Documents. The presentation is actually Power Point and the documents are in Word.
I left Python and Pandas and the visualization was far behind what R had in its tool chain. I still feel that R does visualizations best.
RStudio (A true IDE) was better for my work flow than IPython/Juypter. I actually really missed IPython at first and than slowly I grew to love RStudio.
Not a wall, maybe a paper cut. A 1 index (R) over a 0 index (Python) is pretty smart for doing statistics, less errors.
I also liked the flexibility to use whatever I wanted to do my work. Was difficult at first than learned to love the fact that there were 3 good ways to do something.
Functional Programming. You can do some good functional programming with R. Functions are first class citizens.
Biggest reason for anyone to switch is Hadley Wickham and his work with RStudio. It really is amazing everything that he has done for R in the last 5 years.
Once again Python will get the job done but these are some of the reasons why I switched.
When you say "flexibility to use whatever I wanted to do my work", do you mean data munging and manipulation or statistical modeling packages?
Graphics I can use just about anything. I can use plottly, ggplot2, D3.js, PDF report, HTML, send to a blogging site, interactive graphics with ggvisand Shiney, even PowerPoint slides.
<-- Maturity -- Fundamentals well designed -->
R Python JuliaI have fallen in love again with Python (2) for this reason. I tried 3 but I don't need all it's accountant-like seriousness. It's less fun to code in. When I need to design for servers or for performance I will do golang or ocaml. When I need simple scaffolding for numerical work, and scripting for prototyping integration of various systems, Python 2.7 is like an old, unfussy, friend, helping me along my long and winding data wrangling road with a minimum of cognitive tax. It just works. Long may it live.
In my view the Python powers should look at numerical / data science and throw all of Python's weight behind that, and forget about the web, where they've already lost.
I code in both every day and just don't see what the fuss is about. To me, they're barely different dialects; more like different accents.
Then all this u"xx" stuff on strings. What's that about? I don't care about unicode. 256 ascii characters is fine for me. If I need Unicode I can do it, but I don't need it by default.
Iterators instead of ranges. Fine. Why again? To save memory? I've got 32 billion bytes. I don't need it. It's complicating something to please the computer scientists and it's messing with my mind which has far bigger problems to solve, so it's unpragmatic. If I see htop moaning about memory, I can easily change my code.
Generally, I just like the 2.0 attitude. It's carefree. It just works. Three is trying too hard. Python is just a tool for me. My really hard problems are my daily battles with nonconvex optimization of vast data sets.
I felt this way before I started spending all of my time on web apps. It's reading user input data from some random public source, like Twitter, that forces it upon you. Then, so quickly it became the best practice to "unicode all the things". I think of analogous to how we store timestamps in UTC always.
OTOH: Which characters do and don't belong in unicode and in what order? I don't fucking know. :-)
Should we use decimalized time or time based on the Babylonian base 60/12 system? Both have clear advantages. I don't fucking know. :-)
The world has standardized on Unicode, which (as a collection of expanding standards) defines the set of valid code points and their order. There's still some debate as to UTF-8 vs. UTF-16LE (and perhaps UTF-16 w/BOM and UTF-32) encodings, but Unicode has clearly won. It's not perfect, but it's silly to pretend Unicode hasn't won.
Source: I used to work as an engineer on the content converter portion of Google's indexing system, which took the world's web pages, PDFs, etc. and converted them into a unified format (the text portion of which is encoded as UTF-8) for the rest of the indexing system. Sure, we saw some percentage of EUC-KR, GB2312, Big5, and Win CP1252 text, but Unicode has clearly won and UTF-8 and UTF-16LE are steadily replacing all other encodings.
But plenty of people were writing code that conflated bytes and strings and broke in subtle ways on systems with non-English locales. Maybe you don't notice the bugs, so you don't care, but that doesn't mean that it's not a major flaw in Python 2. Here in Asia, plenty of people may be cursing your silently buggy Python 2 unicode-naive code.
It's similar to a pair of time-handling functions in a domain-specific language I use at work sometimes. The lazy way of converting Times to Strings (arguably reasonably) uses the current processes's time zone to render the Time. Likewise, Time's constructor uses the current processes's time zone when constructing a Time form a String. Someone decided to write a pair of functions that shift a given time forward and backward by the current processes's time zone's UTC offset on the day in question. The one function gives you a Time that if rendered using lazy String formatting (usually) gives you the same String you would have gotten if you had used a function that took a time zone as an argument and passed UTC as the time zone. The inverse function allows you to take a String representation of a UTC time, pass it to Time's constructor without specifying a time zone, and then shift that time so that you (usually) get the same Time you would have gotten had you specified the correct time zone in Time's constructor. This sounds absolutely insane, but it mostly works, except in corner cases that cross DST changes. (Note that 2014-03-09 02:31 America/New_York just doesn't exist, but 2014-03-09 02:31 UTC is a perfectly sensible time.) We now have a jslint-like program that issues a warning if you use either of these functions. Freely guessing at the correct conversion back and forth from byte arrays to code points is very much like this insanity of mostly-working functions to shift times to make their conversions to and from Strings mostly work without having to specify time zones in the Time or String constructors.
Similarly, if your language silently converted April 74th 2014 to June 3rd 2014, it would "just work" for 99.9% of use cases, but it would also hide some bugs. People would complain if the language's designers changed the language's next major version to stop accepting such non-sensical dates. Silently converting back and forth between bytes and code points is more subtle, but similarly insane in hiding bugs that 99.9% of the time would go unnoticed.
Sure, silently ignoring locale conversions of times and strings mostly just works, but that doesn't mean it's not terribly broken. PHP rightly gets a lot of criticism for often following the philosophy of "do what I probably meant, and it will just work 99.9% of the time and cause insane bugs 0.1% of the time".
Python 2:
map(func, iterable) # returns a list
map(func, iterable) # returns an iterator
list(map(func, iterable)) # returns a listI feel like that will be quite an ear-plugging, eye-covering endeavor.
And that's really the point: it has the potential to create a community in which half the people are eyes-closed, ears-plugged.
On your second point: Yeah, I don't mind casting things to a list. But it does seem... I guess, dramatic.
This is a big concern of mine as well, both before/after listening to Guido's keynote on Type Hinting at PyCon (https://www.youtube.com/watch?v=2wDvzy6Hgxg).
At first I thought type hinting would only really see adoption in performance-oriented sections of code, but I think we're all curious to see what best practices emerge in general, and in the open source side of things. Really the additional robustness it unlocks for IDEs like JetBrains is what got me really excited about type hints in Python.
Agreed. But that's only useful if the syntax is non-awful.
What are your thoughts on the future of Python + NumPy vs. MATLAB or R? It seems the numeric and scientific community is embracing Python more, but what I've heard is mostly anecdotal.
I have less experience with Matlab for cost, and lock in reasons. I need to have the option to distribute my code easily.
If you look at Python adoption from the perspective of someone picking it up for the first time, the very first issue they encounter is the "which version of Python should I use" debate, which pretty much frames the experience from that point onward.
The pragmatic attitude that's kept Python 2.X afloat is a key part of why Python has had longevity but it also doesn't set a great tone for "changing the world" or keeping up with the latest sexy fads
We're still not seeing any mainstream Python 3 adoption (no, I don't count your own project). PyPI downloads was still < 5% last I checked. The big web frameworks are still better tested and better performant with 2. Scientific computing is stuck with 2. Same with Ansible. And the big companies you mention have no (public) intention to migrate.
The core developers decided long ago that 3 is what everyone should use, so I have no doubt that will be the case. The open question is how much is left of the ecosystem by then. It's commonly discussed at conferences but there is no straightforward answer. At least we got the TLS stuff backported to 2 which might be indicative of a coming thawing.
If you introduce a new developer to Python the first step is "ignore Python 3, nobody uses that" and then explain that there's infighting about backwards incompatibility so most packages either don't support it or do it badly. Why do I have to justify to new talent the reasons why I'm using a language that can't even agree on the latest version?
I think mainstream adoption is increasing, but not as quickly as anyone would like. The sentiment around adoption I felt echoed at PyCon this year was: "we have dependencies that aren't ported to 3 yet [... and who will be the ones to port them?!]".
I predict that 2.7.9.x.y.z will continue well past 2020 and backport any truly useful stuff from 3 because that's much easier than porting 2 code to 3. Or, and the core developers should seriously think about this, they should build 2->3 right into the interpreter so you can run, unmodified 2.7 code on a 3 interpreter (2.7 libraries included), and then pick and choose the 3 features you want yourself.
False. NumPy, SciPy, scikit-learn, Pandas, and friends all work great on Python 3.
Python 3 won't see any traction unless it offers a clear improvement over 2. Some people think that's async. I have my doubts.
For instance, the split between Python 2 and Python 3 has kept from me getting more into Python because I have to choose one or the other in terms of which libraries I want to be compatible with (and the libraries are a big reason I'd want to to use Python.)
If you look at Java or PHP or .NET there is a lot of concern about keeping compatibility between versions, but going from 2 to 3 they changed a lot of APIs that they didn't have to.
The most disturbing thing is that I have been shouted down for years on places like proggit for bringing this issue up but it is only in the last year that I have seen it talked about by pythonistas, and that's a general problem I see with Python, that it ducks the hard questions which Java, in particular, faces before anyone else. (i.e. having a memory model for threads which is mature)
On one hand, the GIL means that Python is not scalable in terms of threads. Today 4 core machines are mainstream, even for phones, and with a language that supports scalable threads, you can get a 4x speedup easily for many real apps.
The other option is go to the Node route, which is balls-to-the-walls async operations in a single process, but (i) the async people also have had a culture of "kicking the can down the road", (ii) it is hard to get programmers who are used to sync APIs to switch to async (this is one reason why the "Modern" interface in Win 8 was DOA... Microsoft was pushing people to use Javascript because rewriting Win32 apps to be async defeats the purpose of using Win32 which is being able to use all the old stuff and not learning anything new.)
Multithreads in Python works just fine for I/O bounded operations. Switch a multithread program to use multiple processes (and avoid GIL issues) is quite simple with the multiprocessing module[1] if you need to achieve 100% CPU usage.
In my experience, complains about the GIL are mostly FUD.
Clojure's core.async can pull off the same trick, though I don't know as much about how it works.
The GIL sucks. It holds Python back from wider use. I wanted to use Python for the GUI/scripting layer of a high performance desktop application, but dealing with the GIL in a mixed C++/Python multithreaded event-driven architecture was too annoying. When Python code isn't in charge of the main application loop, things get ugly.
These days, I mainly see Python as a source of syntax inspiration for other languages. I say this with the highest amount of respect. I think Python's best qualities - list comprehensions, generators, itertools - are largely independent of its interpreted, dynamic nature. They will eventually find a better home in a language with more industrial-grade semantics.
"The GIL isn't that bad, get over it" is not an attitude that will sustain a language for decades.
There are exceptions. Django rest framework, for example, is an extremely mature, well-made library. But these are becoming less of a percentage of the whole and are harder to find in the chaff.
And there are a few projects still carrying the torch--but typically these are old projects that have been around for a long time, still maintained by older developers who were writing Python when Python was young. We still have at least a decade before these old masters start retiring, so I don't think Python will die any time soon. But it has become more important than ever to be careful what you let into your dependencies.
Though I haven't used either myself yet, I've noticed the trend in automated code quality tooling becoming more popular in Python. Specifically, the ones that come to mind are: Codacy, Code Climate (both free for open source); and Landscape.
Personally I'm still using strong classics like coverage and pylint, but plan to explore the new gen stuff soon.
Strangely, the Google V8 team repeated Python's API design error and also relies on global interpreter variables and thread-local variables to swap interpreter contexts, or "isolates" in V8-speak. Although my understanding is that the V8 team recognizes the problem and are (slowly) addressing it and changing their APIs accordingly with each successive release.
I know undergrad CS students, masters CS students, non-CS professionals well into their career, and non-CS hobbyists. And they are all writing Python. Five years ago when I was doing my undergrad, that would have been Java, or C++.
It's not exactly an exciting change from a HackerNews viewpoint, but it's pretty amazing for programming as a whole. The people I know are so much more excited to be learning. There is frustration, but much less than when I knew people learning on C++. Obviously that's still a fantastic language, but I'm happy Python is the new default.
Given, the quality of things in any community repo, and the namespacing, are often terrible, but this hasn't ever kept me from using good things out of it.
I'm also not sure how it can be a step backwards, as Python is quite a bit older :)
One thing I don't like about npm is folks are starting to store non-npm things in it, which I think may because of the weird situation of npm getting funding. I don't understand that just as much as I wouldn't understand PyPi or CPAN getting funding :)
"Things I wish Pip learned from Npm" https://medium.com/@alonisser/things-i-wish-pip-learned-from...
One feature I'd love to have personally is a clear way to manage only direct dependencies, but have reproducible builds in pip. @nvie has done some great work in this space [0][1], but not all of us have caught up yet. :)