Reducing Python String Memory Use in Apache Arrow 0.12 (2019) | Hacker News Reader

Reducing Python String Memory Use in Apache Arrow 0.12 (2019)

arrow.apache.org

91 pointspolm2314 comments

Relatedly, perhaps the most impactful performance optimization I let some of the science side know about on my team is to declare columns as categorical in this same basic instance. The memory and speed gain is impressive.

It would be fun to compare the two. Interesting that the blog author is the creator of Pandas and didn't include a comparison.

Simple scheme for deduplicating strings in Python:

    dedup = {}
    for s in strings:
        s = dedup.setdefault(s, s)
        # Do something with s

It hashes each string "s" and looks in the dictionary "dedup" to see if there's an existing string object previously in the list "strings", in which case it uses it, and otherwise it stores the current string object in "dedup".

Simpler scheme for deduplicating strings in Python: use `sys.intern`. It also specifically increases hash performances because interned strings can be compared by address after a hash collision (and furthermore hash caching means interned strings almost certainly already have their hash computed and stored)

    for s in strings:
        s = sys.intern(s)
        # Do something with s

Depending on your application lifecycle you may want to shove the dedup'd strings in a set (or in a list, after checking if the identity of what you got from `sys.intern` is different than what you put in) in order to keep them alive: `sys.intern` does not do that.

Interestingly according to docs this doesn't prevent your stirngs from being GC'd either.

"Interned strings are not immortal; you must keep a reference to the return value of intern() around to benefit from it"

> Interestingly according to docs

Also the second paragraph of my comment.

dedup = set(strings)

While this works for dedup'ing a single set of strings, it doesn't work for dedup'ing string identities across the application lifetime because you don't have a fast way of fetching the deduplicated version from the base.

Then what do you replace this line with?

    s = dedup.setdefault(s, s)

There's not an obvious way to get the object in a set that matches a value you have.

I use something like:

   @functools.cache 
   def dedup(s): 
       return sys.intern(s)

Isn't sys.intern a cache itself? What does adding another layer of cacheing on top add?

You're right, PyUnicode_InternInPlace (which is called by sys.intern) already caches results using a regular Python dictionary, so @functools.cache wouldn't add any value.

It happened a long time ago, but I think I misread something about sys.intern not being able to handle lots and lots of strings, so I added an extra layer of caching, but it looks like it's not needed anymore.

[1]: https://github.com/python/cpython/blob/3.10/Objects/unicodeo...

There should be "string deduplication" in the original title. That would be informative and targeted more to Arrow users.

2019

Reply on news.ycombinator.com