PyPy 4.0.0 Released – A Jit with SIMD Vectorization and More | Hacker News Reader

PyPy 4.0.0 Released – A Jit with SIMD Vectorization and More

morepypy.blogspot.com

303 pointsmattip144 comments

The good news never stops with these guys. PyPy is so good that I've stopped considering myself a "Python programmer" to a "PyPy programmer" over the last few years. Best of every world. I get my preferred language and eliminate CPU performance worries.

For me, targeting development on PyPy has taken the place of migrating to Python3. Which just moves too fast in effort to pull people over, Python is too big for that strategy. At this point, people want speed with a great language, they want PyPy! The PyPy team has pretty much enabled the perfect development stack. I use PyPy+Django+gunicorn+Nginx and no problems.

Keep up the great work.

I'll go one step further and advocate for the PyPy developers please not to port to Python 3. Here we have a project which has clearly become the go to for 2.x people who would like Python to move forward in a way which makes a real difference to them, and there is an ideal opportunity here for this to become the de-facto continuation of the 2.x line, with a real future, as an alternative to the 3.x forcefeed.

Not a bad idea. It would certainly elevate the project to unforeseen heights and probably turn them (http://www.baroquesoftware.com/) into very wealthy men while offering a very desired and needed service for many businesses. While harming no one who didn't already harm themselves. More relevant than CPython3 overnight, which is of significant stature.

If prestige and money are desirable things, absolute no brainer if they have a business bone in their bodies. It would also secure the future of the stack for everyone concerned with Python3 threatening to cut off Python2. It could also enable a bigger return on those of us who have invested financially into PyPy development. It would certainly snowball even more financial investment.

It would then be the CPython3 project's move. Abandon and integrate all the non-breaking Python3 features into 2.7? Or just move forward with a significantly weaker and less relevant project, working against their past selves? I don't think most would really care and it's best they just move away with their own project.

Could be on the way judging from the favorable naming convention, PyPy4? Has a nice ring to it.

"the Python2.7 compatible release — PyPy 4.0.0 — (`what's new in PyPy4.0.0?`_)"

"the Python3.2.5 compatible release — PyPy3 2.4.0 — (what's new in PyPy3 2.4.0?)."

The PyPy gods are in an enviable position for sure. If they seize the opportunity it's the chance of a lifetime.

Yeah, the best thing they could possibly do right now is to fork Python and make yet another flavour. We already have this 2/3 mess so let's throw another version into the mix just as Py2 seems to be dying.

I don't get this hate for Python 3. Yeah some things changed, yeah having a big codebase in Py2 sucks if you don't have the time/resources to switch, but starting new projects in Py3 by default is awesome. Asyncio and async/await are fantastic and worth the switch alone.

Yes, I agree that Python 3 is not that bad to start a new project in. But that's not the problem. The problem is that many of us have a huge investment in Python, and the project is meandering around, borderline-insulting its historic users who still like 2.7, while simultaneously failing to deliver on REAL needs such as performance and GPU support. Instead we get these little tweaks here and there, downright embarrassing PEPs like type annotations, and all this frankly distasteful pressure to move to 3.x using threats of the end-of-the-line for 2....

Basically 3 gives us lots of headaches, very marginal improvements, an ivory-tower sense of "we know best" (message to Guido - no you don't), and a real concern that the powers that be have no clue what's going on at Golang Towers, Fort Javascript, not to mention Clojure, Julia, or even Elixir.

They are already willing to take money to work on Python 2.7 stuff, how does refusing to take money for Python 3 stuff lead to them making more money?

It would show leadership. Leadership which is demonstrably lacking in the 3.x line. Thus they could take a (risky but calculated) bet to take over Python completely.

What would be the point of this? Are you suggesting PyPy forks the Python language itself? PyPy hasn't implemented new core language features, and I don't see them doing so in the future. Python 3 is actively developed, fixing many warts in the language and adding great new features.

As annoying as the transition to Python 3 is, it's the future of the language. I was actually one of the curmudgeons who stuck with 2.7 and grumbled about not wanting to switch until a few months ago. I've accepted that I need to, at the very least, start new projects in Python 3 wherever possible. The vast majority of popular third party libraries have been ported over at this point. Python 2's death is inevitable, even if it will take many more years.

That is exactly what I am suggesting. There is no urgent necessity to add any new core language features to 2.7. That's what you 3 guys just don't get. I certainly don't need asyncio as much as I need vectorised numerics, and if I needed asyncio, I had at least 5 libraries already. I most definitely will never be type annotating Python. Cloud cuckoo land.

Let me warn you. If you persist with your propaganda on Python3, then GOLANG will be the future of Python. Does that register with you guys?

Nobody who uses Python 2.7 all day every day thinks it has any of these imaginary "worts" you 3.x guys like to pitch incessantly.

> Nobody who uses Python 2.7 all day every day thinks it has any of these imaginary "worts" you 3.x guys like to pitch all day.

Maybe people who use 2.7 all day don't see the warts as much as someone who has been rid of them?

It took me a week struggling with Asyncio to wrap my mind around it. It took me 12 hours to do the same with Golang channels.

I got 20 000 inserts per second from a 360 gig JSON file into Cassandra using 30 goroutines. I was maxing out at 900 in Python 3. Golang was parsing the JSON at a frankly RIDICULOUS 2 million lines per second. Python 3 was around 50 thousand.

Python 3 is adding features that other languages do better. Python smashes other languages in numerics, and approachability. Nobody needs 3 for numerics, and 3 goes backwards in approachability. If you're using 3 for its back-end tools, you're using the wrong tool.

I would never move to golang for numerics because pandas/numpy/scipy/theano stomp all over it, but its only for numerics that I'll stay with Python. Everything else I'm moving away (mostly Golang, looking at Elixir) and talking to the numeric stack over zeroMQ/msgpack.

> 20 000 inserts per second from a 360 gig JSON file into Cassandra using 30 goroutines. I was maxing out at 900 in Python 3. Golang was parsing the JSON at a frankly RIDICULOUS 2 million lines per second. Python 3 was around 50 thousand.

Newsflash, Go is fast. I'm sure those numbers will be worse in Py2 than Py3 as well, but aside from that what's your point? Python is slow so... don't switch to Python 3?

> Python 3 is adding features that other languages do better. Python smashes other languages in numerics, and approachability. Nobody needs 3 for numerics, and 3 goes backwards in approachability. If you're using 3 for its back-end tools, you're using the wrong tool.

Sure, people using it for numerical stuff might like Py2, but is that the only use case of Python? Should I not switch because you do numerical stuff? Django is great for backend stuff, as is the whole ecosystem (twisted, flask, DRF, asyncio).

Python 3 is also far more approachable than Python 2. Try explaining to someone why their program exploded after typing an umlaut into it, the magical "from __future__ import division" incantation or even the silly super() syntax.

Newsflash. Python 3's main pragmatic selling points are Unicode and Async. Golang is Unicode by default, its concurrency model is much easier to reason about, and it's 20x faster. Why on earth would you use Python 3 (or 2) for your backend? You like buying 20 server boxes when 1 golang box will do? If you think Python will survive as a production language on servers, you're dreaming. It's done, defeated, history on the server outside of legacy code. Django? gimmeabreak.

So what's left that Python does better than any other language? Numerics. That's the only place where you still have a sticky user base that is not likely to dump Python. So don't be so quick do dismiss it. It's the only leg Python is left standing on that other languages cannot easily devour.

Approachability. Not much in it. For every 3/2 = 1 you cite I give you unintuitive lazy ranges, the complete idiocy that 01 != 1, and the unnecessary print functionization which removes one of Python's greatest first-line simplicity selling points. But I will give you that Python 3 remains approachable as a first language. However on that front, inroads every day from JS, where a new coder can get "wow" results in the browser much more quickly than the boring ol' VT100 terminal.

Julia has multithreading and channels. Better syntax than go, more like python. Better for numerics...why use anything else?

> Let me warn you. If you persist with your propaganda on Python3, then GOLANG will be the future of Python. Does that register with you guys?

This abstract future of Python has already been ceded to something that is not Python, just as once the abstract future of Perl was Python. Ruby once tried to take the mantle but failed. Go is now taking a lot of mindshare (from 2 or 3 more I wonder?) and I agree the Python 3 crowd just makes Go more likely to succeed, which is terrifying since it's probably the worst possible choice. My message is simply despair and accept the inevitable: Python will fade -- by all means keep using it, people still use Perl, scientist will be using it for a long time -- but the rest of the software world will pass you by. If you care about what the rest of the world is doing at all, you can help make sure the future passes to better general purpose high level languages like Nim or Clojure or OCaml, and for specific niches like scientific computing goes to the likes of Julia eventually. Anything but Go.

I'm currently registered on the Ocaml mooc ;-) Very intrigued by Nim too.

Yes I find Golang to be dry and uninspiring, but as per one of my other comments, I get 20x the performance, and the concurrency model beats Asyncio hands down. Honestly I needed to get 360 gigabytes of JSON into Cassandra and Python was going to take 6 days (there is some light conditionality on each datapoint preventing a raw dump). I took 1.5 of those days to learn how to do it in Golang, started it at midday today, and estimate it'll be done when I get into work tomorrow morning. Sometimes I just need to get stuff done.

EDIT: Just re-read your post and cannot help but agree that Python's demise is already telegraphed. History. Perl->Python->(?) is a nice analogy. Python won't die for sure, but it will not flourish like it did in the past decade. And yes I think it's good advice to look to more ambitious languages than Go, even if Go solves many "today" problems quite well. Personally wish Clojure would come off the JVM but am looking into Ocaml, Elixir, and when I need imperative, good ol' C wrapped up in Nim. All good candidates, though I really wish something would come up and go "fully vectorized" for the brave new world of GPUs. Personally need a REPL which is why Ocaml wins for now though Spark looks to me to be the "new R" for data science so I shouldn't exclude Scala (JVM notwithstanding). Decisions decisions...

I wonder how much faster pypy would be in parsing 360 GB of data.

It will be interesting to see what the final winner of GPU programming looks like. There's also the possibility of FPGAs making their way into consumer hardware, and right now the pure-Python library MyHDL is a great choice for those. There are a few attempts at abstracting over OpenCL like https://github.com/eholk/harlan, and some intriguing domain specific attempts like http://halide-lang.org/ but I think it will be a while yet before something takes over. I also don't think whatever it is can succeed without corporate backing.

Python Numba compiler (fast as julia) has a fair amount of Cuda and AMD HSA capability.

You're looking at this from the SV startup echo-chamber perspective.

People use Python because the language gets out of the way fairly quickly to let you get things done fairly quickly. Sure, it isn't a rigorous engineering language, but the ratio of small projects that just need something good enough and easy to work with to large/complex projects requiring careful engineering is staggeringly large. While Python is definitely cooling off; people have been working with it for long enough to want something better - unfortunately there just isn't anything on the radar that is significantly enough better to overcome Python's third party library momentum at the moment.

Believe me, I'd love a language with a really flexible/optional functional type system, better metaprogramming facilities, a better concurrency design, slightly more consistent syntax, etc - just as long as it is still easy to think in and convenient to use like Python.

> Believe me, I'd love a language with a really flexible/optional functional type system, better metaprogramming facilities, a better concurrency design, slightly more consistent syntax, etc - just as long as it is still easy to think in and convenient to use like Python.

As someone who, among dynamic languages, dismissed Perl quite a while back as not as useful as Python and Ruby, I'm starting to think Perl 6 may be the language that hits that spot.

No please! Julia Julia Julia fills all those, but with beatiful python like syntax, lisp like macros when you wnat them, go like channels, distributed capabilities and leading class numerical stuff. Also webserver with a front end that you can build with polymer in Pure Julia.

http://escher-jl.org/

Julia!!!!! The answer to all these problems, both for general and science programming.

I would suggest toning down the arrogance. Just because you don't personally need some features does not mean that people who want the language to progress and not stagnate are idiots.

That's rich when it is the 2.7 users like myself who are constantly bombarded with arrogance and condescension. Nobody wants Python to stagnate. The argument is which direction it should take.

I use Python 2.7 all day every day and I definitely still notice the warts and find them irritating. They are minor warts, and almost never affect my productivity, but they're definitely things that should have been, and are, fixed. super(what do I pass here again?), old vs. new style classes, integer division, input, print, exception bubbling/chaining, nonsense compares between different types, True/False/None as regular variables, no "nonlocal"... Then there are the features I can't use, like extended unpacking and merging, better yielding, function annotations, and lots of improvements to the standard library in terms of their API and functionality.

I'm not a fan of asyncio, but the other stuff is quite awesome. It's not compelling enough to port some of my older Python 2.7 projects, and isn't a revolution of the language or anything, but they're good improvements. And even if they weren't good improvements, Python 3 is the future, and Python 2.7 simply won't be a viable option for a lot longer.

You contribute the same borderline FUD in every thread that is related to Python. It's unfortunate that Python is not the language you want it to be, but consider just how dismissive you are of the needs of others - frankly, it comes across as arrogant.

FUD? I have >100k lines of highly sophisticated numerical analysis Numpy/Python code in fixed income which has cost literally millions of dollars. I'm not going to let that investment get toyed with by the webified 3.x crew without vigorously stating my point.

Could Python 3 host the features you want? If it did, would you consider migrating your code-base to it? I'm positive that if your ideas have merit and you share them on the python-ideas mailing list, you'll get a good discussion out of it at the very least.

> you 3 guys ... Cloud cuckoo land ... your propaganda ... Does that register with you guys?

Your comments have been crossing into incivility. Please don't do that; it breaks the site rules and leads to tedious flamewars. I'm sure you can make your argument civilly and substantively if you try.

I was overly aggressive. It's regrettable. But every point I made has been seriously contemplated, and is in my opinion worthy of consideration. The extent to which the 2.7 community and its arguments, are treated with derision by the 3.x people, makes some form of vigorous defence sometimes necessary. I'll add that unlike a large silent group of Python people who basically agree with me, I am not afraid to spend karma points speaking up against the received wisdom unambiguously and forcefully.

Now that I have made all my points, it's bye-bye.py. I won't comment on Python again.

> It's regrettable.

That makes it sound like you had no control.

Being rude is not "speaking up against the received wisdom", it's just being rude, and it's against the rules here. In the future, please post civilly and substantively or not at all.

https://news.ycombinator.com/newsguidelines.html

https://news.ycombinator.com/newswelcome.html

Kudos to the pypy team! I suspect pypy would benefit from some kind of marketing campaign. It rarely lets me down and yet lots of python devs don't know about it.

The most popular package(s) that don't work with it have been numpy, scipy, etc. When I first encountered pypy, there was a numpypy and I wouldn't be surprised if that's different now. But bottom line, tons of python code out there needs more performance and has no dependency on numpy or any C extensions.

we're trying to address both numpy and C extensions, stay tuned ;-)

Keep up the good work!

you should kickstarter this - and also the python 3 support.

we did kickstart the python 3 support on our website. Donate!

Link: http://pypy.org/py3donate.html

> But bottom line, tons of python code out there needs more performance and has no dependency on numpy or any C extensions.

What they need is nice JIT engines like PyPy and not to switch languages.

If no one invested in improving implementations for modern languages and switched to something lower level all the time, we would still be using Fortran for business applications.

I think you guys are saying the same thing. Ie, because they don't need to switch languages, and PyPy is a JIT engine for Python, these tons of Python code would benefit from if the devs were aware of PyPy.

Indeed that's what I meant and perhaps pjmlp thinks that the feature set in numpy could have been / could be implemented in pure python, now that we have PyPy.

What I mean is that I see PyPy very positive, as a way to stay in Python instead of jumping into C, Go or whatever language.

In terms of performance naysayers only get convinced when someone proves them wrong, not because of what we think might be possible.

So it is important that we have research in GC systems, JIT compilers for dynamic languages, optimizers for FP and LP languages and so on.

Otherwise we might as well keep on using just Assembly.

Numpy is way faster than what you would get if you implementing things in pure Python and executed with PyPy. That's why there is numpypy in the first place.

Devs are already aware of PyPy. If it was as easy as being aware, everyone would switch tomorrow. The problem is that tons of code uses C extensions created with swig or python boost and those take a lot of time to port to PyPy. Another problem is that PyPy is sometimes slower/less memory efficient than CPython when you wrote the code that is optimized for CPython. That is abusing dictionaries and very limited OO use.

PyPy has higher memory overhead for very small units of work but usually once you do more than a couple of seconds of work it usually uses significantly less memory than CPython. As far as I'm concern who cares if a Hello World program uses more memory or some small trivial few lines of code. I'm more interested in real world code which my experience has been that PyPy uses significantly less memory than CPython. If you know of a case where this is not true report it as I'm sure the PyPy devs will consider it a bug.

PyPy also has a slower startup time/overhead so very small amounts of work may end up slower than CPython but in general larger chunks of work will be faster if not its a bug and should be reported.

There is currently some work going on to improve the C extension issue. It's currently in its beginning stages and its goals are to provide both a significant amount of compatibility with existing extensions and a significant reduction in the overhead to call out to C extensions over what is currently implemented in PyPy. On the performance side the new approach was said to remove about 40% of the existing overhead. There are tons of corner cases that still need to be worked on for the new approach to become production worthy and there is no guarantee that there will be no show stoppers that come up or that the tail of corner cases that need to be dealt with becomes a nightmare and the approach is abandoned. If this works succeeds, it will be a game changer as suddenly the majority of libraries would then become compatible with PyPy.

We still are using FORTRAN for Business Applications and I fail to see the need to switch them to a slower language to modernize them. Unless the switch is solving a program leave them alone. This is why in R there is still plenty of FORTRAN code.

Personally I LOVE Python but I always feel it is the 2nd best choice. I do use Python but it rarely is the best tool to use for the problem you are solving. Maybe Pypy will turn this around, and I hope so.

I don't understand numerical analysis as business applications.

For me business applications are what you would use Cobol, Clipper, Java, Delphi, C#, Eifel, ....

Of course Fortran makes sense for number crushing, but that is the language domain, not doing CRUD, ETL or distributed computing stuff (not counting MPI here).

Accounting, spreadsheets, transaction processing, business intelligence?

Of course those are business applications as well.

Do you intend to refer to a product done in those areas Fortran in modern days, maybe with Fortran 2008?

Outside scientific computing and heavy number crushing algorithms and libraries, I don't see a use for it.

No, I'm just saying they're business applications. A segment of business applications are numerically focused. Fortran handles them well. So, in that segment, it can be considered a language for business applications with numeric focus.

I'm not saying anyone should use it today lol. Julia and R are the best for that sort of stuff. Way the hell ahead of Fortran. Although I thought about making a nice web application in Fortran just to screw with people who eventually try to extend it and gasp in horror. ;)

http://www.aim.env.uea.ac.uk/aim/info/fortran.pdf

R != Business Application.

Used R as an example of a language that still uses FORTRAN. BUT I think Microsoft and Business Analysis people would not agree with you.

How can SciPy be fast if it is written in an interpreted language like Python?

Actually, the time-critical loops are usually implemented in C or Fortran. Much of SciPy is a thin layer of code on top of the scientific routines that are freely available at http://www.netlib.org/. Netlib is a huge repository of incredibly valuable and robust scientific algorithms written in C and Fortran. It would be silly to rewrite these algorithms and would take years to debug them. SciPy uses a variety of methods to generate “wrappers” around these algorithms so that they can be used in Python. Some wrappers were generated by hand coding them in C. The rest were generated using either SWIG or f2py. Some of the newer contributions to SciPy are either written entirely or wrapped with Cython.

A second answer is that for difficult problems, a better algorithm can make a tremendous difference in the time it takes to solve a problem. So using scipy’s built-in algorithms may be much faster than a simple algorithm coded in C.

http://www.scipy.org/scipylib/faq.html#how-can-scipy-be-fast...

This is also why the scipy/numpy crowd doesn't care much about PyPy. In this world Python is the equivalent of ducttape. You don't performance optimize ducttape, you want it to be easy to use and to fix any problem you have.

PyPy is taking a similar approach. The difference is Numpy is using CPython's CAPI as the duct tape where as PyPy is using cffi as the duct tape. Both approaches end up using the same underlying libraries.

Too many people just have the wrong impression, thinking that PyPy plans on re-implementing all the libraries that Numpy and scipy use but that's just completely false.

They have re-implemented the Numpy array so that it can take advantage of the JIT and so that parts of an algorithm implemented In Python that uses Numpy can also be optimized. Unlike what occurs when using Numpy under CPython where the Python code does not get optimized unless it is converted to Cython, C Code, or some alternative to Python to have it be optimized.

I like the quote: "Python is generally the second-best language for everything".

I am a Java dev, I heard of PyPy a few years ago, I find it amazing and often tell people about it when talking about performance and other related topics.

The thing is I've never found another developer that new about it, something is very wrong...

If you are a Java developer and find PyPy amazing, you should also look at Truffle and Graal http://lafo.ssw.uni-linz.ac.at/papers/2013_Onward_OneVMToRul..., http://chrisseaton.com/rubytruffle/, https://bitbucket.org/ssllab/zippy/src/700f88dc2a7d2011258ae...

Is PyPy going to support Python 3? We kind of need the ecosystem to move forward, and Python 3 has a lot of niceties...

Yes - it's getting there and you can help it get there sooner by donating http://pypy.org/py3donate.html

Doesn't it already? 3.2.5 it seems.

http://pypy.org/download.html

Also I recently discovered that pypy supports non-ASCII characters pretty well which at least doesn't work with my version of python 2.7.6

In what way?

In variable names, for example. You can do this:

    >>> π = 3.14

In Python 2:

    >>> π = 3.14
      File "<stdin>", line 1
        π = 3.14
        ^
    SyntaxError: invalid syntax

  Python 2.7.10 (850edf14b2c7, Oct 29 2015, 17:32:05)
  [PyPy 4.0.0 with GCC 5.2.0] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>>> π = 3.14
    File "<stdin>", line 1
      π = 3.14
      ^
  SyntaxError: Unknown character

Are you sure you weren't trying it in Python 3 (or PyPy3) rather than PyPy?

I'm getting same result as scardine:

    Python 2.7.8 (default, Sep 30 2014, 15:34:38) [GCC] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> π = 3.14
      File "<stdin>", line 1
        π = 3.14
        ^
    SyntaxError: invalid syntax
    >>>

also:

    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> π = 3.14
    >>> print(π)
    3.14

You are right, this is Python 3 stuff. I just assumed the parent was talking about Pypy3.

> We would also like to thank our contributors (7 new ones since PyPy 2.6.0) and encourage new people to join the project. PyPy has many layers and we need help with all of them

This doesn't match my experience of the PyPy project. I found a tiny bug in the stdlib matching against CPython, went into IRC to ask a question about test running to be sure I got it right and was quickly engaged in conversation about why I was running core tests. Next thing I know my small bug has been fixed by a core developer and my chance to contribute is gone.

I'll stick to being a user.

Is this a complaint or a compliment to the developers?

He didn't want to report a bug, he wanted to contribute to the project. I can think of a few reasons why. They lost a new potential committer by "being pragmatic" about it, but ignoring inter-personal relationships, like so often.

Personally I'd be quite excited to see a bug I reported fixed the same-day - of course people's expectations are different it's not always easy to balance these things.

One of the rewards of participating in OSS is to make your name visible, this can then be used as creds to gain commit rights or to talk at conferences as well as proof of competence when looking for a job.

Interestingly, I've had the same experience when reporting and asking how to contribute to Python 3, not by a core dev but somebody "stealing" my bug and quickly submitting a patch!

The open source community is not there to provide you with cred and visibility. It's there to so software can be developed in a way that certain people think is the best way to develop software.

Something similar happened that left me with mixed feelings recently. I recently came up with a new feature idea for Python, and a core dev ran with it. I was very happy at first, but then he wrote a pep, implemented it, and got all the congrats. I had hoped to help, and put something on my resume, but was shut out. "Thanks for improving my pep" on a mailing list is all the credit I got (and work I was able to do on it). Kind of disappointing, though I will enjoy the feature.

Sometimes person reporting an issue along with the code, doesn't get it quite right and core dev can immediately see correct solution and can't make himself to wait until original reporter makes it right... of course it doesn't happen often but it does.

One of the project I participated the most was ansible, and in a case like this, @mpdehaan2 would just push back and comment until your PR was acceptable, which was encouraging and making you contribute more.

I can understand that, I certainly would have been stoked about a bug report being fixed on the same day; I hadn't reported the bug though. I was working on providing a bug report with a patch alongside it and (perhaps foolishly) mentioned the bug informally as context as to why I was doing this.

Without being privy to the entire context, what I think might've happened is that your discussion caught the interest of a core developer who then picked it up and ran with it. I doubt they intended to step on your toes as a consequence of fixing the problem. Hackers tend to be intrigued by problems, including small ones, and especially ones that cause software behavior to deviate from expectations.

The other side of the coin is that without a patch, the developers have no idea what your motives or capabilities are. They can't read minds, so if you pique their interest sufficiently, they're very likely to take it upon themselves to do the legwork. No malice intended. I should hope you'll change your mind about contributing to FOSS projects after reading this thread, because I think it's counterproductive to take this sort of thing personally (I don't know if you are, but it appears to me that you're unhappy about this at the very least).

That said, there aren't many projects that post credits or thanks to people who have discovered bugs or whose line of questioning has lead to fixing application behavior. While it's extra work and isn't always feasible, it can encourage users to pitch in if they know their name might appear in some credits (even if it's only per-release). Then there's the question of how big the problem needs to be to credit someone...

I'm a long-term FLOSS contributor, but I've also served on the boards of 501(c)(3)s, I've been a mentor for Google Summer of Code for most of the times the programme has run and I've run workshops at my local university about getting involved in open source. I know what is involved in making a community welcoming to new contributors. If a project is putting an explicit call for contributions in updates that implies they're not getting as many as they'd like, which fits my experiences.

What you say about hackers getting nerd-sniped is true, but "that's just how coders are" certainly isn't. If you're a core developer for a project and you want the community to grow rather than diminish it's your responsibility to make it socially welcoming as well as having a low technical barrier to entry. I personally have spent many hours responding to emails about problems I could have fixed trivially, but if I'd done that the potential new contributor may not have grown into someone that's regularly fixing bugs I can't fix trivially. I'm not taking the unwelcoming attitude personally, I'm acting accordingly. If I can't get the information from a community I need to participate but I can get them to do the work by just throwing it over the fence I'd be crazy to continue to try and help them unless I have a very strong reason to outside of requiring fixes.

As to your point about reading minds, this is specifically mentioned in my initial comment. I asked about a problem running tests, and when I provided the context that I was asked for (I'm trying to put together a patch for this bug) there should have been no question as to motivation.

> I hadn't reported the bug though.

I don't think its reasonable to expect a projects core developers, when they become aware of a bug through mechanisms other than formal bug reports, to deliberately stall confirming and resolving the problem because the person raising the issue is entitled to a crack at fixing it.

You can do something correct and still be discouraging to people who would like to contribute.

How is reporting a bug not a contribution? Now, I don't know about this particular bug, but in most cases finding the bug is where the work is. Checking in the fix is trivial.

That's a shame, it can sometimes be tough to pick out a good manageable intro-bug on an open source project you want to contribute to. Perhaps it was a new-ish core dev who was also keen to cut his teeth on small/doable bugs?

>This doesn't match my experience of the PyPy project

One experience, with one developer, and without going through official channels to asks for participation etc.

The IRC channel is an official channel, in fact it's by far the most important one of the PyPy project.

Did you tell them that you wanted to contribute, rather than get a bug fixed?

On the flip side of this thread, some projects have started to adopt specific tagging in their issue systems for folks looking to get started as contributors. I really like this as it's clear communication to both new contributors and to existing contributors that the issue in question is an on-ramp for the project.

Here's Neovim's "entry-level" label, for example:

https://github.com/neovim/neovim/issues?q=is%3Aopen+is%3Aiss...

As another example it seems that Rust does the same:

https://github.com/rust-lang/rust/issues?q=is%3Aopen+is%3Ais...

Servo inherited that policy too: https://github.com/servo/servo/issues?q=is%3Aopen+is%3Aissue...

Actually pypy tries to do the same, bitbucket does not support tagging very well though (but looking for "easy" or something in the issue tracker helps). I am personally sorry if the poster had an experience like that, IRC is difficult to communicate over, it must have been a misunderstanding

This is very very disingenuous.

Here are the public IRC logs for what I assume you're referring to (which is from July 2014 by the way):

https://botbot.me/freenode/pypy/2014-07-02/?msg=17362202&pag...

You found a bug in an old version of PyPy. A core developer told you you might want to try a new one. No one stole your shot out from under you.

You would undoubtedly be welcomed to contribute. Don't post FUD.

Disclaimer: I am not a PyPy core dev but I do work closely with them.

> Alex_Gaynor - simpson: import operator; operator.isNumberType is the check you want

> I'll push this change quickly

> Alex_Gaynor - uhh, wait the check seems to exist?

I dunno, I kinda see both sides here. First, I don't think the devs were off-base at all here. They were friendly and looking to solve problems. However, they did seem to want to simply solve the problem, while Matthew did state that he was looking at this as a way to contribute.

The whole interaction could have been improved with a little encouragement at the end, and possibly a recommendation of an 'easy' bug that was currently in need of a fix. It was fairly clear that there was an enthusiastic new contributor, so a little effort in this direction may have been warranted.

I certainly would consider this an example of being unfriendly or unwelcoming.

I apologise that I mis-remembered and characterised intending to fix a bug, investigating, and finding out it was already fixed as just fixing a bug, however I believe the rest stands.

The fact that I wanted to run tests and couldn't, and the team went on to why I wanted to run tests and looked to solve my problem is my complaint.

Also, to be clear, I'm not trying to say the PyPy channel is unfriendly, it certainly isn't. It's just not set up to help potential new contributors get started.

Not everyone can get it right all the time.

Fixed a bug the same day someone asked about it and some potential contribute blame you for not letting them fix entry level bug.

Answering people with "patch welcome" and people blame you for not fixing it.

My experience with PyPy was that it was faster for mapreduce jobs without JSON parsing and slower for JSON-heavy jobs. Because, I assume, CPython benefits from a C implementation of simplejson and PyPy doesn't. Does anyone know if this has changed? (I'll probably install 4.0 and see for myself, but curious if anyone knows what I should expect.)

We did put some effort into making built-in json faster. If you're using an optimized version like uJson, it'll be faster under cpython so far, but it all depends whether there is more processing or more parsing involved. We're happy to help you with benchmarks though

I've only used simplejson. A mapreduce job could be reading billions of lines and possibly doing very simple logic on each line, so the parsing/encoding is a big percentage of the processing, sometimes nearly all of it. I'll try 4.0 and see how it does now.

Try builtin json instead of simplejson for pypy (it's API-equivalent)

On big jobs (millions of json input/output) I have found ujson significantly faster than json + pypy.

yop, as I said, ujson is probably your best bet if all you do is parse json.

If you are on python3 python-rapidjson is faster and safer than uJSON:

https://github.com/kenrobbins/python-rapidjson

    >>> data2 = 1.23456789e-13
    >>> rapidjson.loads(rapidjson.dumps(data2))
    1.23456789e-13
    >>> ujson.loads(ujson.dumps(data2))
    0.0

One has to be careful with strings in pypy, because they don't have a string-builder optimization. Or maybe they do now?

I am curious. Usually in numpy basic linear algebra are delegated to BLAS and potentially really fast if configured to use a fast BLAS implementation (OpenBLAS, MKL, etc). Does the numpy fork of PyPy use the same approach ?

yes, but BLAS only helps you with some operations and not others. In fact most of "basic" operations are still done "by hand" in some templated C in numpy. Additionally, doing vectorized operations lazily (which is not present right now in pypy but was at least considered in the past) yields great improvements.

sounds great. I have a scientfic library that relies on a mix of loops and matrix multiplication. I need to find some time to test it with PyPy

Have you also checked out Numba? JIT and type inference, but native to CPython and designed for numerics:

http://numba.pydata.org

yes. However the delegation only happens if you use the numpy.linalg module, or if numpy has chosen internally to call the numpy.linalg module (i.e. logic in numpy.dot will call out for large ndarrays)

Our vectorizing JIT can use SIMD semantics on all numpy looping calls. For instance, non-matrix multiply A*B or for ndarray + scalar calls

While many numpy users are in the habit of manipulating large square matrices, there is a significant number of users who use small arrays, or process RGB pixels

It's hard to tell from these posts under what circumstances will SIMD vectorization kick in outside of numpy. E.g., will it work for plain old list comprehensions, which are everywhere in my code? That would be awesome.

Generally speaking "no", but there are a few cases where it might, e.g. if you're using arrays/numpy arrays in list comprehensions. We could not get it to run on normal lists for a bit obscure reasons.

I did a small test, here's the gist:https://gist.github.com/arnists/a9f19af5d9469549020e

Seems like a huge improvement, don't know if it's the recent additions or in general pypy vs cpython but it's enough of a speed bump to sit up and take note.

That's why you don't use pure python for this. Here is your test done using numpy

    import numpy as np
    %timeit sum(np.sum(np.random.randint(0, 10000000, 5000)) for i in range(5000))
    1 loops, best of 3: 586 ms per loop

I was mostly getting a comparison between cpython and pypy, less concerned with numpy vs cpython.

Sure, but my point is that it's not a relevant comparison.

A relevant comparison is counting words or something like that. Things people don't have an easy way to do much faster.

> We would also like to thank our contributors (7 new ones since PyPy 2.6.0)

That's very good news, since monetary contributions don't seem to be abunding (none of the goals have been met according to pypy.org). Maybe it's a marketing issue, like said before?

Thanks for all the effort made.

python 3.4 please ;_;

The value of PyPy is that it is a thing you can just drop in to make your code faster. If PyPy wants to gain any traction they pretty much have to concentrate on the version with the most existing code.

most of my existing production code is 3.4, hence my question.

I'm/we are also moving everything to 3.4, since we want all the nice things.

Python 3 is the future. There's a chance that when Pypy finally catches up with Python 2.7 (numpy, C extensions etc), it will be obsolete.

Python3 is a language. Python2 is also a language. Python3 might someday become wildly popular, but that doesn't magically make Python2 go away just because the name sounds the same.

From [PEP 373]:

> The End Of Life date (EOL, sunset date) for Python 2.7 has been moved five years into the future, to 2020. This decision was made to clarify the status of Python 2.7 and relieve worries for those users who cannot yet migrate to Python 3. See also PEP 466.

> This declaration does not guarantee that bugfix releases will be made on a regular basis, but it should enable volunteers who want to contribute bugfixes for Python 2.7 and it should satisfy vendors who still have to support Python 2 for years to come.

> There will be no Python 2.8 (see PEP 404).

Take note that the initial date was set to 2015 [2] and it was delayed only last year.

   [PEP 373]: http://legacy.python.org/dev/peps/pep-0373/
   [2]: https://hg.python.org/peps/rev/76d43e52d978

I too want 3.3+ support but I know some claim that there's very little demand from people actually doing something with PyPy for Python 3 support. I really think it may be a chicken and egg problem: that is, support will generate demand.

In any case the 3.3 branch seemed good when I compiled it last, it'd be nice to see a partial release.

The problem, which they keep mentioning over and over, is that companies who actually donate have zero interest in a python 3 version. While they are a non profit, they have to pay the bills somehow.

Chicken and egg... But if there was good Python 3 support maybe more companies would contribute.

Yes, this is one of those features that should be targeting the newest release, to provide a reason to upgrade, and not support lock-in to 2.7.

"puhleeze give us Python 3...." repeated everywhere in this comment thread, yet the authors of pypy keep responding with the same line: pay some money and we'll do it. This speaks to what the real industry users of Python already know: 2.7 is where it's at, and they're prepared to pay for that, and not 3.x. Sure there'll be a bunch of people waving the 3 flag and saying "I use it" etc, but the bottom line is the bottom line: money. 2 attracts the dough.

If I were the PyPy guys I'd just announce that PyPy was now officially a fork of 2.7. This would bring all the numerical guys with them, all the corpos with big investments in 2 would fund them, and we could then let the 3.x zealots go off on their own tangent.

Anybody who likes what 3.x brings to the table should really spend a weekend learning Go, which does it all better, and faster.

I've been saying in these threads for ages that vectorization is the big win in Python (thank you PyPy), and not hobby projects like asyncio (miles better elsewhere), or (eyes rolling) type annotations.

According to data I'm getting a 20X performance increase from the rendering engine in Django. I know this is the least of the time I spend but it's significant enough to take a deeper look.

You may be surprised, I've seen a lot of Django setups where template rendering was the slowest or second-slowest part of their system behind DB queries.

Still not spending a lot of time there due to proper fragment caching.

I wish it had faster startup time, which really matters for Mercurial.

Working on it (even wrote 2 blog posts recently about it). Mercurial is an interesting benchmark/use case for us.

Yay! Glad to hear it. Using Python makes me so envious of Perl's startup time. Maybe some day we can have that in Python too...

Where are those blog posts?

https://morepypy.blogspot.pt/2015/10/pypy-memory-and-warmup-...

https://morepypy.blogspot.pt/2015/09/pypy-warmup-improvement...

Are there recent analysis/benchmarks of startup time? This is the major blocker preventing us switching to PyPy, since we run a lot of short-lived processes.

Guys, what's a "trace optimizer"? Google is failing me by returning results about MySQL.

Search https://www.google.com?q=trace+optimizer+compilers

According to wikipedia: https://en.wikipedia.org/wiki/Tracing_just-in-time_compilati...

Tracing just-in-time compilation is a technique used by virtual machines to optimize the execution of a program at runtime. This is done by recording a linear sequence of frequently executed operations, compiling them to native machine code and executing them. This is opposed to traditional just-in-time (JIT) compilers that work on a per-method basis.

They're referring to a tracing JIT[1], I believe.

[1]:https://en.wikipedia.org/wiki/Tracing_just-in-time_compilati...

I assume it's a reference to tracing JIT (https://en.wikipedia.org/wiki/Tracing_just-in-time_compilati...).

It's a JIT that traces runtime execution, rather than looking at whole method or function bodies. In particular it eliminates control flow, which simplifies optimisation.

Traces are an artifact produced by some JITs.

Basically you observe loops and loops that satisfy certain criteria are traced, that is all operations performed within one iteration are stored. Such a trace is then optimized, compiled to machine code and then executed for every iteration when the loop is encountered again.

How up-to-date is this? http://doc.pypy.org/en/latest/windows.html#what-is-missing-f...

I've read that a few times and I always come away confused -- it sounds like a huge fundamental type change needs to be made by someone well versed in the PyPy internals. i.e. not something you'd typically defer to an external contributor.

I'd like to experiment with PyPy and PyParallel, but I'm basically exclusively 64-bit Windows, so it sounds like a non-starter.

Is there any work going towards getting pandas to work with PyPy as well? Its the only thing keeping me from using it.

Nice! Does this work with scipy now?

so the next version of pypy3 should be 5.0.x, lol

Reply on news.ycombinator.com