Python consumes a lot of memory – how to reduce the size of objects?

The article doesn't mention a hidden gem in Python's standard library - typed arrays!

Those are in package `array` and they are a very barebones version of numpy's ndarray - with just one (explicit) dimension and no overloaded operators. But if you just want to keep a bunch of numbers in a contiguous array, they can save you tons of memory.

(I know the purpose of the article is to describe a more complex data structure, but arrays can still get you very far.)

It's not always an option, but simply using PyPy can massively reduce memory usage without needing to change your code at all.

The PyPy site links to this blog post (from 10 years ago) with some info: https://morepypy.blogspot.com/2009/10/gc-improvements.html

And a quick search found this relatively recent post that does some measurement of it: https://dev.nextthought.com/blog/2018/08/cpython-vs-pypy-mem...

Any type that you have "a lot" of, is a bad candidate for a class/object. Your OO program should typically never instantiate thousands of heap allocated objects at once. A triangle mesh is a good object candidate. One triangle or vertex is not. A device, a render system API handle and an image is a good candidate, one pixel is not, and so on.

So don't use a lot of objects. In C# you use a struct and arrays of them to avoid creating an object on the heap per array entry. In java you have to resort to SoA, in Python it's the same. Just because you have objects doesn't mean everything should be an object. Even languages that follow an "everything is an object"-design usually have an escape hatch such as plain value arrays.

While everyone haggles about the internals I have an interesting anecdote about the costs of tools libraries.

A small service that landed in my lap needed to read(only) a data source that was roughly 10k lines of yaml. No way that was going to be in any way efficient so I asked for suggestions. All the work-a-day devs(I am not) instantly said the same thing without a single real thought about it: Make it a database duh!

Long story slightly less long, Loading up the libraries to interface with a database ate between 2 and 3 times the memory(depending on the DB and lib) that simply loading the entire 10k line yml ate and offered slower performance and required more code.

SQLite was pretty darn close but in the interest of saving developers from themselves vis a vis parameterized queries, or the need to queries all together for that matter, increased the required code for zero benefit.

The service still hums along with a 10k line yaml in memory. "Worse is better" indeed.

In the pure Python cases the 8 bytes per attribute are just pointers. The x, y and z are themselves full-blown objects with all the extra memory overhead that comes with it and this is not counted in the article. For example, an int object uses 28 bytes, so three of them already use up more than each of the described container objects.

The Cython and Numpy cases directly store the actual data and this has the larger effect to reduce memory.

Many workloads do not require more than 4GB addressable RAM per process. Linux offered a 32-bit user space with 64-bit instruction set:

https://en.m.wikipedia.org/wiki/X32_ABI

As effectively many Python workloads are usually object oriented business logic and objects are mostly pointers, setting up x32 user space "halved" the memory usage. It also made execution performance faster because of better CPU cache utilisation.

Sadly, x32 was "very custom" and very hard to support. Last I heard x32 is being phased out from Linux kernel.

I am a C/C++ programmer mostly. Is there good documentation for python development when I care about performance and memory footprint out there as a book or something that anyone can recommend? It was fun reading this but I like to know more.

>A significant reduction in the size of a class instance in RAM is achieved by eliminating __dict__ and__weakref__. This is possible with the help of a "trick" with __slots__:

What is the downside to the __slots__: trick? In what cases do you need the dict and weakref?

TXR Lisp, 64 bit:

  1> (defstruct blank ())
  #<struct-type blank>
  2> (pprof (new blank))
  malloc bytes:            16
  gc heap bytes:           32
  total:                   48
  milliseconds:             0
  #S(blank)

32 bit:

  2> (pprof (new blank))
  malloc bytes:             8
  gc heap bytes:           16
  total:                   24
  milliseconds:             0
  #S(blank)

The structure instance has a pointer to its type, followed by a numeric ID (which is also in the type, but is forwarded to the instance for faster access). The ID is combined with a slot symbol to perform a cache lookup to get the offset of a slot. The numeric ID is a fixnum, which leaves a few spare bits for a couple of flags:

  struct struct_inst {                                                            
    struct struct_type *type;                                                     
    cnum id : sizeof (cnum) * CHAR_BIT - TAG_SHIFT;                               
    unsigned lazy : 1;                                                            
    unsigned dirty : 1;                                                           
    val slot[1];                                                                  
  };                                                                              

If someone wanted to shrink this, they could patch the code so that the dirty flag support, and lazy instantiation of structs is made optional (as in compiled out), and so is the forwarding of the inst->type->id to inst->id. This struct type is not known outside of struct.c, which is only some 1700 lines long. If you take out the id member from struct_inst, the C compiler will find all the places that have to be fixed up; literally a 15 minute job.

I can also think of a more substantial refactoring that would eliminate the type pointer also. There is no room in the heap object handle to store it directly: heap handles have four words in them; the COBJ ones used for structs have the type field, a class symbol, a pointer to an operations structure, and a pointer to some associated object (in this case struct_inst). All structures share the same operations structure. However, if we dynamically allocate that operations structure for each struct type, we could stick the type pointer in there, taking it out of the instance. Thus an instance size could literally just be sizeof(pointer) * no-of-instance-slots.

I am paying US$5 a month for python's memory hogging.

Running Mailman it will not run in 1GB of ram (the US$5 VPS) so I had to give it 2GB. 2GB? For a mail list server?

Had the same problem running motioneye. It will not run on a Raspberry PI Zero. It is advertised to run on a Zero (Apparently I could squeeze it on by doing some Python magic....)

What total crap is Python! What waste, what hubris, what technical failure!

I suspect the problem is actually Django - what hubris! NIH! lighttpd is running sweetly, with some rust templating in a acceptable fraction of the memory.

Seems to be an error in editing:

> ...which received a rating of [stackoverflow] (https://stackoverflow.com/questions/29290359/existence-of-mu... -python / 29419745). In additioobjects liken, it can be used to reduce the size of objects in RAM...

Is this trading space for speed or is there actually a speedup as well in some cases? With interpreters it's possible you be small and fast if you become perhaps obscure or more rigid in your structured definitions

This is a pretty neat and useful comparison of how much memory different structures use in Python to achieve roughly the same goals.

I always hate how these things are phrased. The idea of “using a lot of memory” doesn’t exist in a vacuum. “A lot” relative to what? Do you require run-time dynamic typing features? Do you want to leverage the Python data model? If yes, then this is the memory cost of your desires.

Advice in many of the comments about other languages sounds so tone deaf. The question is not about using less memory. It’s about getting _exactly_ the feature set of Python while using less memory.

I feel like the last two are cheating a bit by explicitly using 32 bit integers where the other examples seemed to use 64 bit.

OP, if you're reading this, your article has damaged syntax at the word `additioobjects`.

this functions u will also see in the case of JAVA s well.this problem arises due to large no. of objects are active in RAM durig the execution of a progrm especially if there ae restrictions on the total amount of availablle memory