Hacker News Reader

Comment by vegabook | Hacker News Reader

In other words, a lost opportunity. Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!. We're clocking up 3.x version numbers as if this will magically provide the illusion of progress, but the killer feature is still not there. Instead we get type annotations. In a dynamic language. Which doesn't compile and therefore doesn't need them. With no performance advantage. If I have to type declare everything, I want 10x performance. Okay?

If Python were a listed company, the CEO would have been replaced long ago. I'm tired of watching my favourite language flail around like this. Will Continuum Analytics or Enthought please fork 2.7?

View on HN

> Asyncio and this new syntax is both hard for beginners and experienced python coders alike, and still doesn't do multicore!.

The fundamental limitation here isn't unique to Python. Multiple tasks that are CPU bound simply cannot be managed using cooperative multitasking, and that's all asyncio is: cooperative multitasking, with a friendlier syntax than the old asyncore module.

Cooperative multitasking using async I/O works great when you have lots of tasks running that are I/O bound--they spend almost all of their time waiting for network or disk reads/writes. But as soon as you start piling up tasks that require CPU, you need either multiple threads or multiple processes. Python asyncio was never intended to handle that use case.

Multiple threads is where Python has a unique limitation: the GIL prevents multiple threads from running on multiple cores, so if you want multiple CPU bound tasks to use multiple cores, you need to fork each one into a separate process.

It also doesn't help that the multiprocessing module is, IMO, a huge ball of cruft that's overkill for a lot of multitasking use cases. But there are simpler ways to do it, at least on Unix systems, because the fork system call is very lightweight.

Shameless plug: I wrote a library for this some time ago, which I use whenever I have a Python project that needs to fork worker processes; it's the "comm" sub-package in my plib-io Python distribution (this is the Python 3 version):

https://bitbucket.org/pdonis/plib3-io/src

I know it's not easy to graft multicore onto Python. So why pretend? If you're serious about web programming you're not using Python anyway. You're using JS or Golang. And if you insist on python, we have plenty of options to do async i/o already. Optimising async is like polishing the doorknobs on the titanic. Waste of effort.

What we need, is a concentrated focus on the only two areas where Python beats the competition hands down. Numerical computing, and newby accessibility. The first gets a token "@" operator (great, thanks), the second is moving backwards with 3.x. Asyncio, type annotations, and unicode, are answers to questions nobody is asking (except by those who refuse to use the right tool for the right job).

> If you're serious about web programming you're not using Python anyway.

Web programming in general is not CPU intensive, so async I/O (which Python already does well, as you point out--except for the API issue, see below) works fine for web programming.

> if you insist on python, we have plenty of options to do async i/o already

But all of their APIs suck, at least IMO. Supporting async I/O with built-in language syntax, to make code easier to write and more readable, seems like a good idea to me.

I think the limitations of the GIL are not all in all a bad thing, but I do think that the Python community needs to seriously look at making Asynchronous execution a priority in future versions. Asyncio is really complicated and provides very little in terms of performance.

Multitasking is the most important issue that Python faces today (and maybe PyPy is the answer).

The problem that python is facing (and pypy is not solving) is that in order to have a working async library ecosystem, everything, especially the stdlib has to be async aware. This is a far cry right now and I don't see any of the python devs really trying to address that. Otherwise you end up with the same situation as for twisted - it works and it's great, but every single library needs to be made twisted aware

So we're in 3.5 already, and one of its central features, async, is flawed, you say. So not only do we not have a killer feature, but even one of the nice-to-haves is a dog. Timeout. This project needs new leadership. The competition (Golang) is walking all over Python on the async issue.

I read your comments on this thread, and I frankly wonder where you are coming from. It seems you care a lot about Python, but have a crisis because your pet peeve (threading) is not being addressed, and as a result expect the team who has been guiding Python to the success that it is over the last 25 years to step down. (And it has been a huge success - it has none of the big corporate money backing that Java, C# and Go have - but mindshare and notability on the same scale, dominates significant niches like non-HPC scientific computation, and has very significant presence in almost every field).

Could you tell me what (and how long) your experience is?

My answers about your complaints are basically:

1. Solving threading is (relatively) easy, if you just give up backwards compatibility; Not the 2.x -> 3.x compatibility which is comparatively trivial - but in a major way, breaking every single extension library, and the vast majority of Python code (or slowing it down unbearably). You might be happy, but the rest of the Python users (basically, the reason you actually use Python) won't.

2. Assuming they agree with you about their failure (I dont, FWIW), someone has to step up and offer an alternative. Who is that, what are they doing these days, and why do you think they will succeed where Guido et al "failed"?

3. Perl at 28 years is not much older than Python at 24 years (relatively speaking), but has been sliding into obscurity for a long time now, whereas Python is flourishing. The Python 3 transition is actually happening as planned (IIRC, there was an expected 5 year period just for feature and speed parity!). Perl 6 is esoteric, Perl 5 is aging and dying. I think that for a living, popular language, the Python team is doing a commendable job, even if they do not address a specific issue (that many people care about, but would actually not make that much of a difference in practice if we are to learn from other languages)

The GIL issue has been solved without breaking backwards compat: http://pyparallel.org/

It just need further dev work and acceptance into py3

On Windows

Did you actually look into it? The threads you run can't make any change to existing objects, and various other changes. It does break compatibility, and needs patched version of Numpy, ODBC (and I would guess, most other packages).

Definitely not "solved without breaking backwards compat".

I have been using Python for a decade, and intensively for 7 years. I am a domain expert (finance) but I do also have 4 years of CS so consider myself non-idiotic, if not haskell-genius, on programming languages. I adopted Python (before it was popular - it was behind ruby / PHP / perl at the time) because it was pragmatic above all. That last statement is being violated with 3.x, and that is the seed of my concern.

Before answering your questions 1 thru 3, some context. It is my strong belief that the authorities are constantly looking at golang and JS as their competitors, in other words, the web world, whereas the real hardcore advantage of Python is in science and numerical computing. As evidence, witness numerous Python books which advise new users to hit the Continuum Analytics or Enthought sites for their full-stack Python installations, even texts which are not about numerics.

On your questions:

1) I don't care about threading. I care about pushing as many compute bits through the Xeon as I can in a given amount of seconds (using Numpy). But as I am a data scientist, I need the REPL. C is out. Why can I still not do this easily? Multiprocessing is there, sure, but it's been unch for years, while Cuda, OpenCL etc are far too hard for guy like me whose intellectual bandwidth is occupied with the domain, not the CS. Isn't that what Python was supposed to be about? Getting stuff done? Why isn't Python vectoring my data through the CPU and GPU yet, 15 years after numeric was first introduced?

2) Continuum Analytics is doing an awesome job and I don't see why they, or Enthought, couldn't take up the mantle, 10gen/Datastax style to use a database analogy. They really know their customers, and the Continuum stack delivers real new value every 6 months, and not only for a scientific audience. More generally, real users in real domains should be driving the project.

3) I am less concerned about Python 3 happening as planned, than I am about the focus of the project. Type annotations? This is an intellectual indulgence if it does not increase performance. Asyncio? We've had async libraries for years! Even when I started Python we had async libraries (not as good but they were there). How is async something fantastic and new? It's nothing but polishing an existing capability a little bit further. Unicode. fine. But again, web focused. Nobody else cares. Xrange laziness. Okay. Leaves me cold. Print(). No CS benefit, but huge marketing loss as you can't go out to newbies anymore and say "hey, check this out... Python hello world?"

  >>> print "hello world"
  "hello world"

It just doesn't get any simpler, and yet Python wants to throw out that unique hook for new people who care little about coding but a lot about domain. It's zero, genuinely zero, boilerplate, whereas there are a dozen languages where you can do print("hello world"). Seems trivial, but in print vs print() we have the key difference in philosophy (get-stuff-done-now vs take-me-oh-so-seriously). If you're so serious at a computer science level, you're not going to do Python.

So. What is Python. A serious language? NO. A wonderfully malleable, not too serious, friendly language, into which you can insert some real hardcore stuff (Numpy, ML, website parsing, database transformations, game scripting, image processing....the list is endless) really easily? Yes.

Where is Python genuinely way ahead of everyone else? Only on numerical computing.

It strikes me that Guido and co are embarrassed by their weekend hack of 1.x and 2.x, when that is precisely what the user base loves about it. Their attempt to make Python serious, is killing Python's original spirit. There is nothing wrong with 2.7. Nobody wants Python to morph into Java.

So, am I a Luddite wanting 2.7 to live forever? no. What I want is vectorization plus DAG-like workflows. These are the most important pieces of computer science that actually dovetail with real world use cases, today. Yes async is cool, but golang now owns that space. What I'd really like is for Python to give us a good framework for the Big Data world which is in almost everybody's use case now, and that means, Python needs to talk multiprocessor, Python needs to talk GPU, Python needs to talk cluster, and Python should long ago have been addressing this directly. Python needs to "go vectorized". That should be the project's obsession. SIMD, in a word, where the parallel granularity can go from GPU kernels,to CPU, to multi-machine clusters, and DAG-like workflows (with possible recursion) built in. This is not a nice-to-have capability anymore. It is what the next wildly popular mainstream programming language will have, built into the language. The signals from everything we're seeing added to 3.x is NOT this. It's web-like stuff. That battle is over. JS won.

Let's not gift the opportunity of huge data to Java (Spark) and Cuda, or a new language that will see the future better than us. It should be Python!

There you go. My view.

> Where is Python genuinely way ahead of everyone else? Only on numerical computing.

No, that's not where it is. It's in being a user friendly language for people who need to get numerical work done, and there's a huge difference. IPython/Jupyter did not happen in any other language, not even remotely (the closest thing I've seen is some Lua graphic repl that didn't go far).

> Nobody wants Python to morph into Java.

Python is not morphing into Java, far from it. 3.x represents a cleanup of the language (str vs. unicode, new vs. old style classes, a lot of other stuff). I do not agree with all of it (I too disagree with demoting print from statement to function), and there is other stuff I would have done - but most of it was desperately needed.

> What I want is vectorization plus DAG-like workflows. These are the most important pieces of computer science that actually dovetail with real world use cases, today

For a tiny, tiny part of the users. And they already have e.g. Theano, and a few other tools, that give some solutions.

> What I'd really like is for Python to give us a good framework for the Big Data world which is in almost everybody's use case now,

Excuse me, but I have to disagree anyone other than a tiny (perhaps 1 percent) of Python users care about big data. Most definitely not "everybody's use case". And the minority which actually cares about big data should not get anywhere close to Python - the runtime is prohibitive.

> This is not a nice-to-have capability anymore. It is what the next wildly popular mainstream programming language will have, built into the language.

Interesting. What language has this today? I know not of a single one, mainstream or niche. I don't believe that you are right on this.

> Let's not gift the opportunity of huge data to Java (Spark) and Cuda, or a new language that will see the future better than us. It should be Python!

Check out Nim + Nimborg. You'll get the best of Python and Nim, and Nimborg would let you do the transition smoothly.

Your idea of Python is not Guido's idea of Python, and as far as I know does not have popular support anywhere. If I were feeling so strongly about my favorite language being so misguided, I would look for a new language. The momentum of existing development direction is essentially unstoppable, whether it is right or wrong.

I fail to see how asyncio should use multicore. Asynchronous programming uses a single thread...

If you use CPU-bound tasks, using non-blocking I/O (which is what asyncio is about) is not going to work. You can use multiple processes (maybe from concurrent.futures[1]) to help that case.

[1] https://docs.python.org/3.5/library/concurrent.futures.html#...

They pretty much already did, but with 3.x: http://pyparallel.org/

Amen brother.

If you make one mistake in your thinking, it's treating Python as more than a christmas hack (which it started out as). It's not a serious language, it has no design behind it and is decades behind the state of the art.

Use it for throw-away scripts and such is what I say, it's an area where it shines.

The moment you want to _architect_ something substantial, reach for something else cause chewing gum and duct tape is not going to do it.

Amen buddy. I hack in Python. I discover in Python. I script, and I pre-production in Python. Then I do C or Java or even JS. But none of my pro-code would have been possible without all the learning I accumulated about the problem in good ol' 2.7 interactive and its arsenal of tools.

[edit] Actually I think you're right. I think it's time to graduate to something serious. My mistake is to think that Python can be that serious language. 3.x, by trying to be that, is horribly compromising the basics of what Python is all about, namely accessibility, friendliness, discovery, malleability. Not production.

We are pretty serious about the web, infrastructure, and platforms in general, and we use Python. As does Dropbox, Google, Facebook, Twitter, etc.

I don't think concurrency is great in Python, but you're living in the wrong world if you dont think people are using the language for real things.

Not sure what you mean by pretty serious about the web, infrastructure and platforms. To me this reads as an oxymoron because Python is not a pretty serious language. It's a toy at best. There is not a single feature that Python introduced or pioneered in the computing arena. It's a mismash of bad ideas from other similarly bad languages, wrapped in user-friendly paper. In that regard, it's very similar to Perl, but a lot more dangerous.

Unlike Perl, there is nothing immediate in Python to warn you of the pit you're digging for yourself, especially if you're a newcomer to programming (as a substantial chunk of the Python community is). The troubles begin when you make the colossal mistake of trying to use Python for more than it can do, try to design and architect actual systems with it but I digress.

On the other point you made, Google, Facebook, Twitter etc are using C++ too.

Does that make C++ a good language for architecting substantial systems? Show me (1) system of substantial size implemented in C++ that hasn't turned out to be a DISASTER in anything _but_ performance. When your browser can be taken over a million ways to Sunday _just because_ it's implemented in C++, do you think Google, Facebook, Twitter etc give a damn if _you_ don't? Same argument for PHP and Facebook.

Would you say that these companies are "pretty serious" about web, infrastructure and platforms?

There are good languages out there, suited to architecting systems that are maintainable, secure and evolve gracefully. You won't find them simply by looking at what Google, Facebook and Twitter are using. You will need to try harder.

Reply on news.ycombinator.com