Hacker News Reader

Comment by yongjik | Hacker News Reader

There's "not for performance-critical use", and then there's "why is it gobbling up all the RAM (~10GB) and make it impossible for me to do anything, when the entire data I fed it is several hundred MB."

Python frequently saunters into the second territory. Well, I guess there are tools available to profile memory usage and stuff, but if I had to spend that much effort tracking down memory issues, I might as well rewrite it in C++. (It doesn't help (or it helps?) that I'm much more comfortable with C++ than Python. YMMV.)

View on HN

There are some easy to avoid pathological cases such as concatenating numpy arrays in a loop.

Take this gem from a well known course:

   np.concatenate([x.next() for i in range(x.nb)])

That looks pretty innocent but it can eat up your memory in an eyeblink if the input is large enough. That's the sort of pitfall that a lot of python code suffers from because the abstractions are just nice enough to make you believe this will work without penalty and without knowing how it is implemented under the hood you're suddenly out a few gigs of ram.

If np.concatenate accepts any iterable, removing the brackets is what you want to do there.

That won't help. Concatenate needs to know the total length up front. It will evaluate all the input items at once.

My experience is that Python is much better about this than many other languages in its class, Ruby in particular. It's going to be hard to beat manual pointer/memory management in terms of raw performance, but Python is pretty respectable for an untyped, memory-managed language.

Reply on news.ycombinator.com