Hacker News Reader

Comment by IgorPartola | Hacker News Reader

A more general question: why is it so typical for a Python we worker to have its memory grow infinitely with each request processed? I don’t expect the kind of performance I can get out of C with its ability to only allocate memory on the stack or to pre-allocate all the buffers I’d need. And I have found a number of memory leaks in Python libraries and especially poorly written Python C modules over the years. But still I would expect something like a simple ideomatic Django application to not have its memory footprint grow idefinitely. Is this just life now? Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed? Are corporations people?

View on HN

> why is it so typical for a Python we worker to have its memory grow infinitely with each request processed?

It's not. The cases where it happened to me during the last 10 years and dozens of web projects spanning on multiple frameworks were generally me doing something stupid:

- using a mutable object as default value or in a class attribute

- doing something in __del__

- keeping DEBUG=True for Django (which is very well known for causing memory leaks)

> Is this just life now?

Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen.

> Are there any good tools out there for figuring out what part of the application keeps requesting objects that don’t get destroyed?

Yes for course: http://tech.labs.oliverwyman.com/blog/2008/11/14/tracing-pyt...

I found pdb to be useful when I wrote my own code from scratch. I found it useless when using a large app on top of Django. Oh really? I have a lot of dicts and tuples? Who knew. I would rather see something like a count of object allocated but not released at the end of each request cycle and their reference graph. Because when I have found memory leaks it has had to do with some kind of cyclic dependencies or creating objects that some kind of long lived object like a database adapter wanted to hold onto.

You got muppy for that: https://pythonhosted.org/Pympler/muppy.html

Also objgraph: https://www.darkcoding.net/software/finding-memory-leaks-in-...

If you are lucky to use Python 3.6+, you got tracemalloc: https://docs.python.org/3/library/tracemalloc.html

Thanks. Those do look excellent.

It's hard to call DEBUG in Django a "memory leak" when the "leak" is documented behavior -- when DEBUG=True, Django keeps a running per-connection log in memory of database queries issued. For, y'know, debugging :)

This does cause memory use to increase with the number of queries issued, but a debug log is not really what I'd call a leak (and genuinely leaking memory in Python takes some effort).

Who keeps a running, unbounded log of all connections in memory?

I’m not a pythonista, so please forgive my ignorance here, but: is this just the default configuration for Django? Does it ship out-of-the-box with adapters to log that to MySQL or similar instead? Why does it not log the last n connections for some sane (and configurable) default value of n instead?

With django, you don't need to set up a webserver or mysql. You can just run `./manage.py runserver` and it will run in it's own webserver and use sqlite.

That is what the "debug=True" variable does. Sets things up for running in a minimal development environment. Another feature? It restarts the server when any source files change.

Suffice to say, it's really not meant to be run in anything like production.

https://docs.djangoproject.com/en/dev/ref/settings/#debug

>Never deploy a site into production with DEBUG turned on.

>Did you catch that? NEVER deploy a site into production with DEBUG turned on.

>Still, note that there are always going to be sections of your debug output that are inappropriate for public consumption. File paths, configuration options and the like all give attackers extra information about your server.

>It is also important to remember that when running with DEBUG turned on, Django will remember every SQL query it executes. This is useful when you’re debugging, but it’ll rapidly consume memory on a production server.

I don’t know. If I were designing that system, I’d note that most queries would be extremely similar and would benefit insanely from being compressed in situ or would (given were talking DB here) be a perfect fit for normalization.

You mentioned SQLite - a second “querylog.db” with a table of queries and a table of instances (query time, parameters, whatever) would be an obvious option (at the expense of slightly exaggerating your db response times, but I’m sure debug mode is already slower as is).

When running in debug mode, generally the server is restarted every few minutes at worst. As the app is under active development and the server restarts whenever the app changes.

The reason it logs those queries is just for debugging. There are a lot of different ways they could do it, but they don't matter because you can just turn that feature off, or set it up to log in a different way, or whatever else.

Logging to a DB would be worse, because then it stores a bunch of useless queries that all look very similar. That logging is used for debugging, and wouldn't be something you'd want to use in production.

Why hasn't anyone already done this? Because it's trivia, not an actual problem that needs fixing.

As others have noted, the behavior is there for use in lighter-weight debugging/dev situations when you want to interact with the code and also have the ability to inspect what it's doing in detail (either by directly poking things from inside a Python interpreter, or through plugins like the debug toolbar). Trying to design a hyper-efficient compressed query storage mechanism would be massive overkill for such a thing. The simplest thing that could possibly work for this is an in-memory list of the queries as strings, so that's what it does.

Since no one has mentioned yet, it IS bounded to the last 9000 queries by default.

> Who keeps a running, unbounded log of all connections in memory?

It does that only when DEBUG=True, which is not the recommended production configuration.

> Nope. I have big Django apps running (like 500k users/day streaming video sites) and it doesn't happen.

Mind sharing your stack? I'm currently torn between Laravel and Django. It won't be the first time I've used an MVC framework (CodeIgniter, some Rails) but I like working with Python. However, Laravel also seems to be heavily used. I don't have anything against PHP but wanting to compound my familiarity with Python is one big reason for leaning towards Django...

Varnish, nginx, gunicorn, Django, redis, celery, postgres most of the time.

Sometime a bit of crossbar.io.

Laravel is a nice framework. I prefer it to symfony, espacially since they love Vue.js which is now my fav front end lib.

But the definitive advantage of using Python is that you get a much more versatile language than PHP. Python is not just good for the Web. It's heavily used for scripting, automation, sysadmin, data analysis, GUI, pentesting, etc.

People don't just use Python with HTTP in mind. You find it at Apple, NASA, at Sony, Fedora, in the state of Geneva, in french schools, in 3D (Maya, Blender) or geography (most GIS)...

Basically, you get a tool that you can use for a lot more stuff, because the ecosystem has been developed for a lot more types of tasks.

My default stack is almost exactly yours, except

> gunicorn

Any particular reason you use it over uwsgi? I last did real research in 2013 and have not revisited since then.

Ease of use. Uwsgi is faster, but it's not the place in my stack where i need speed. The cost of transforming nginx packets in wsgi object is very small compared to all the rest.

I use uwsgi when I wanna use the whole box. Now with k8s use gunicorn

>- using a mutable object as default value or in a class attribute

Python noob here. Can you expand on that a little? Or perhaps link a post? I can't think of an example or how it'd impact memory

Here[0]. Mutable objects as default values create a single instance of the object. For iterables (list, set, etc) the behaviour is particularly surprising coz it accumulates/memoize every item ever added (i.e a list that grows/leaks with every call to the function). If you use a good IDE (like pycharm), it will warn you about this stuff.

[0] http://docs.python-guide.org/en/latest/writing/gotchas/

It would only lead to a memory leak depending on your usage, but it’s pretty common gotcha that can be annoying with or without memory issues.

The following is some pretty good reading on the matter, and the example given is also one that would lead to a memory leak:

https://stackoverflow.com/questions/1132941/least-astonishme...

> A more general question: why is it so typical for a Python we worker to have its memory grow infinitely with each request processed?

I can't talk about general situation but in this specific case it's because they've turned off GC previously, so growing infinitely was expected. Details are in the first blog post they've linked from the OP.

Notice that even with the GC on, even with the modified GC, the memory footprint doesn’t stabilize. It just grows slower.

The default Python allocator experiences a lot of fragmentation over time, leading to sparse memory packing.

Dropbox implemented a custom allocator with different types and sizes getting their own arenas, leading to long term memory stability. Even controlling all their own code, processing millions of file paths, metadata, and buffers inevitably lead to memory leaks.

Reply on news.ycombinator.com