Hacker News Reader

Also of course, like any other language, Python is not immune to I/O wait. Recently I was using Python to do some low level audio processing (nothing fancy) and wondering why it was taking so long to process such a small file. Turns out the "wave" stream reader I was using isn't internally buffered, like most of the normal file readers are, so each single-byte read was making the round trip to disk. Simply reading the whole file in one go sped up the program's execution more than 10x.

I like to think I've been doing this long enough to not make such silly mistakes, but when you're focused on a totally different area of the problem, things like this are still quite easy to miss.

I'd argue that these things are comparatively easy to address, as one use of `strace` will immediately show bad IO patterns like that, and it works the same way across all programming languages.

> Python is "slow", yes, but often the bottlenecks are outside of python, so it doesn't matter anyway.

It matters when it's just Python, onstensibly the only concern, when talking about Python performance.

> So really, if I rewrote it in C++ using the most blisteringly fast JSON library I could find, it would just mean the program will spend more time blocking on gcov's output.

Until the day when clang starts outputting gcov at 30x the rate GCC does it, and then fastcov will become the bottleneck. That said, I'm an user and it is much better than lcov so thank you for it :-)

Very true... and let's hope that day does come, because it means we can generate coverage reports even faster (if not with my tool, with someone else's)!

Agreed, that sounds like a good problem to have!

Node's json parser is so fast that I've seen people recommend using it over static initialization of object graphs as recently as last week. I had to check the date, thinking that I was looking at some post from 2016. Nope, spring 2020.

As an aside, have you benchmarked running 1.5x or 2x gcov instances per core? Most old code is written as aggressively sequential, so any stall causes it to sit doing absolutely nothing while waiting for an operation to complete. The throughput curve for parallel tasks often has a peak somewhere the interval of 1 < n < 3. When n > 2 it's time to have a heart-to-heart with the authors.

Code gets reused, even if you don't plan for it. Prototypes go to production. Chunks of code from one project, where performance doesn't matter, wind up in another project where performance is important.

For this reason, starting with slow code (either Python or just lazy OO-heavy C++) sets you down a path to future trouble. There will never be time for a rewrite, and if you think there is time then a case of "second system syndrome" may be what stops you.

There is also the matter of personal habit. You use what you are comfortable with. If you get comfortable writing slow code, then you'll start projects with slow code by default. You'll hit the above problems far more often than somebody who is in the habit of writing fast code by default.

No offense, but that seems like bikeshedding to me. At what point do you stop making premature optimizations like that? The commenter had a need, wrote a package to fulfill the need in a language they were comfortable with, and got the job done. Performance is important to the project and the performance goals are met.

Why should they have spent more time to accomplish the same goal and same performance?

I think the parent post also may be making an assumption that C++ code takes little more effort/time to write than python. My experience is that this couldn't be further from the truth. I recently did some DSP work, and prototyping it in python then rewriting the code in C++ easily saved me calendar time as well as effort and headache. Python code has a tendency to just work for numerical applications and it would have been a bigger headache to check my work in C++ without answers that were easier to verify in python (I remember reading years ago a thread on benchmarking that got shut down hard by someone pointing out that only the python code was giving the correct answers due to various floating point issues). On top of that, I needed to use templates to get more compile time substitution to squeeze out performance, and struggling with templates and the type checker would have been a context switching nightmare without having the algorithms already working.

For your use case I believe the standard is to write in Matlab, and then use a tool that converts Matlab code to C.

That's hardly the standard. In 2020 you will write your code in Python using numpy and or numba and your done.

I process 4 concurrent HD video streams in Python in real time on a 4 core embedded device.

How much time is actually spent in the python runtime/interpreter? That python can glue some libraries together at a high level was never really in doubt.

How does that matter in any meaningful way ? Are you building real-world software or fretting about problems that you don't actually have ?

I'm a bit surprised to hear this, is this a common thing to do in the DSP world in 2020?

Btw I highly recommend using pythran for your use case which apart from speeding up python significantly allows you to translate your python code directly to C++. Here is a blog post exactly about this use case: https://serge-sans-paille.github.io/pythran-stories/pythran-...