Measuring memory usage in Python: it’s tricky | Hacker News Reader

Measuring memory usage in Python: it’s tricky

pythonspeed.com

116 pointsitamarst14 comments

For folks looking for a good modern profiling tool for Python codebases: my current favourite is Scalene [0], which allows profiling CPU/GPU and memory usage in a performant way. The author gave a talk about it on last PyCon US [1].

[0]: https://github.com/plasma-umass/scalene [1]: https://www.youtube.com/watch?v=nrQPqy3YY5A

Scalene is pretty neat, yes. The memory profiling, however, is more useful when you're trying to find memory leaks. This is a common scenario in web apps, say.

If your goal is to find peak memory usage, which is what you care about in data processing batch processes, Fil is probably a better choice (https://pythonspeed.com/fil).

Having done a serious amount of Python memory profiling work in the past, both Fil and Scalene are useful tools. Add them to your list along with runsnakemem and the high level memory-profiler.

On the time profiling side I highly recommend py-spy and pyflame (a little less updated).

Remember that a lot of software not only relies on memory that the developer explicitly allocated but also on buffers and various caches, disk cache being an example. It goes a long way - kernel might refuse dropping disk cache when it considers it's too hot (ex. db machine.)

The question "how much free memory do I have" or "how much memory does this program use" is a very hard one, and you can't just read the "MemAvailable" column from /proc/meminfo or RSS/PSS/Virt from ps/pmap.

> With this particular measurement, we’re using 3083MB, or 3.08GB

Don't mix up SI and binary prefixes, kids! The listing shows 3083 MiB, which is either 3.01 GiB, or 3.23 GB. There's no mainstream interpretation giving 3.08 of something.

Is MiB/GiB binary? And, GB/MB SI? Or the other way around?

Yes, yes, and no.

It's easy to remember. The SI prefixes are always decimal, that's why people call it "the decimal system". Kilopascals, megajoules, centimetres, always powers of ten. So a megabuck is 10^6 bucks, 1 million bucks, and a gigabyte is 10^9 bytes, 1 billion bytes.

The weird power-of-two prefixes get the special weird symbol and name, because you wouldn't want those anywhere except computing.

I recently had an adventure with Python subprocesses and memory. This isn't just a Python thing, but it's more of an issue because Python lacks proper multitreading: processes don't necessarily return memory to the OS after free(), and Python doesn't necessarily free() unused memory after a GC run. If you're using a process pool where some work can be memory-intensive, this can massively increase total memory use as the job bounces between subprocesses.

If you're using subprocesses, you may also run into the GC duplicating your heap with unnecessary collection. Check https://docs.python.org/3/library/gc.html#gc.freeze and the note about forking.

Good timing! I'm currently battling memory issues with a project where I'm straightening full resolution panoramas. I often reach close to my machine's limit of 16GB.

I think this might help me immensely. Thank you OP.

One random and possibly incorrect memory, from back when I was using scikit-image: switching from order=3 to order=2 interpolation on images can reduce CPU usage and... I think memory usage? significantly, for only minor (but real) reduction in quality of output.

With interpreters it's always difficult to track resource usage. It should have full blown instrumentation w/ kernel support. I am not sure python has it.

Good news! https://docs.python.org/3/howto/instrumentation.html

There are no great tools for getting the common data out yet as far as I know, but the possibility is there and the standard dtrace / usdt hooks are in place. You can use it with bpftrace.

Reply on news.ycombinator.com