Comment by amyjess | Hacker News Reader

Comment by amyjess | Hacker News Reader

Parent Full thread

> It's primarily a testament to how simply mind bogglingly slow Python is outside of its optimised numerical science ecosystem.

From my experience in using Python at my last job, I'll also add that Python is decent at tasks that aren't CPU-bound.

I wrote a lot of scripts that polled large amounts of network devices for information and then did something with it (typically upsert the data into a database, either via direct SQL or a REST API to whatever service owns the database). All these tasks were heavily network-bound. The amount of time the CPU was doing any work was minuscule compared to the amount of time it was waiting to get data back from the network. I doubt Nim or any other language would have been a significant performance improvement in this case.

For what it's worth, that made these scripts excellent candidates for multithreading. I'd run them with 20+ threads, and it was glorious. At first I did multiprocessing, because of all the GIL horror stories, but multiprocessing made it very difficult to cache data, so eventually I said "well, all this is network-bound so the GIL doesn't even apply" and switched over to multiprocessing.dummy (which implements pools using the same API as multiprocessing but with threads instead of processes), and I never looked back.

Edit: For what it's worth, Nim sounds like a really cool language, and it's right up my alley in several ways, I just don't think Python is particularly slow at network-bound tasks that use very little CPU.

I agree with your entire post but, and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that your ETL process could save it into a new table. And boom you‘re into CPU bound lands now. Often it‘s fine, you can wait those 1-2min for a daily occuring process. But what if for example you put this whole component behind a REST API that is behind a load balancer that is set with a certain timeout? There are even strict upper limits if for example you chose AWS Lambda for your stuff.

And suddenly you need to introduce quite a bit more technical complexity into this story that‘s gonna be hard to explain to management - all they see is that you now can insert a couple of millions of DB rows and their Big Data consultants[TM] told them that this is nowadays not even worth thinking about.

Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

>Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.

I somewhat disagree there. IMO you can structure a large application just as well in python as in Java, but it requires more architecture guidance as your degree of freedom is higher. Regarding testing, mypy and later python versions have been a big help there, I don't think it's much of a difference anymore and I'd take it over Node.js for that reason because the core language IMO is more consistent and strict and requires way way less outside dependencies. In fact the standard lib for me is the main reason to choose python today.

So for me it's really performance being the main thing to think about with python. If one anticipates tighter latency or throughput requirements I'd go for something else.

> there‘s often a point where it starts being bothersome

My team and I have been using Python for web, scripting, and ETL development since 2007. I don't recall the last time Python wasn't "fast enough" for anything I needed to do. I'm sure it's legitimately too slow for plenty of use cases and classes of programming domains. But for a general purpose language that makes our developers incredibly productive (which is what we optimize for organizationally), I've not "often" found the point where it becomes bothersome. On the contrary, I'd say I've rarely found it. And in those cases, the workaround to make it fast enough is there.

That's not to dispute your experience. I just want to provide a counter example.

Does Python give you an actual productivity edge? I do not see how.

Python developers are easier to hire (certainly than Nim), have a monolanguage experience (meaning less onboarding), but are not magically more productive. Developer availability is still a great reason to choose Python.

YMMV. My background is in financial applications and there it’s a quite common problem popping up, especially (but not exclusively) in analytics.

Is there a problem with binding c or c++ to your python project in these situations?

The whole build gets quite complex. Numba is interesting but also still a bit limited in what it supports. There are no silver bullets and sometimes you can find a good fit, but IMO the JVM or C#/.net overall have a better story when it comes to write performance critical applications (but don’t need to get the last 20% such as HFT or HPC sims), as you can also find people on the market who understand how to do and maintain that more easily. There are tons of python devs but in my experience at least on European market their understanding of underlying systems tends to be limited.

> "I'll also add that Python is decent at tasks that aren't CPU-bound"

IO-bound tasks are almost by definition outside of your Python application's control. You yield control to the system to execute the actual task, and from that point on - you're no longer in control of how long the task will take to complete.

In other words, Python "being fast" by waiting on a Socket to complete receiving data isn't a particularily impressive feat.

The main point (I think) was that python is a viable language for many use cases that are not processing intensive, while also being very easy and quick to write which is often the most important thing.

In my experience, Python is so slow that it will make CPU bound tasks that have no business being CPU bound.

You have to be reaaaaaally slow to be beaten by a network. Also, touched by the OP, if it takes 3 hours to write some code that in Python takes only 1, or if the compile times are huge, Python can beat other languages in speed (that edge fades when the same program is used over and over again).

But as demonstrated, Nim is fast to write and fast to compile, so Python has little edge. Just it's huge ecosystem.

> Just it's huge ecosystem.

self-contradiction at its best. kindly be reminded the same advantage was the only thing that kept Java alive for so long until it finally started to enter the XXI century a couple years ago.

you just can't discount an ecosystem, especially if its huge.

Exactly. Also I’m not impressed with a lot of the ecosystem. After getting more acquainted with many very important libraries I can see the poor quality and it’s scary. Grateful for the free tools but IMO there other Languages like go where more essential libraries and tooling are written by better programmers. Like actual professionals instead of hobby tier

As other commenters point out, how can a language be fast in a way besides something CPU bound? You are saying it is fast when it's not doing anything. Not sure I understand.

it's a shorthand for 'fast enough'. is it slow if nobody would notice if it was 10x faster?

You might want to look into async (asyncio or anyio) instead of or in addition to threads for network-heavy code. Async coroutines I find can be much easier to debug and develop than OS-threaded code.

Reply on news.ycombinator.com