Hacker News Reader

Comment by Grue3 | Hacker News Reader

>Rows are represented as standard Python dictionaries

That seems inefficient. You're always going to have the same columns in each row, so storing column names for each row is a huge waste of memory. Django's "cursor.fetchall" returns a list of tuples, which seems to be a saner way to represent db rows in Python.

View on HN

Reitz has a public Facebook post on this. Here are his comments:

Harry Percival Looks nice! Y u no namedtuple?

Kenneth Reitz Dictionaries are a much more predictable interface ;)

Kenneth Reitz Just because you can do something, doesn't mean you should!

Well, that's creepy.

Hahaha... I can only imagine. Sorry to creep you out. Your Facebook profile is just a click away from your personal website, and your discussion about Records is up front and center on your feed for me :)

%s/'s/ is not/i Fixed it, you're welcome.

I learned from a coworker recently that namedtuple is a much lower memory usage than a dictionary, so if you're dealing with lots and lots of data regularly, and lots of processes on a single machine, (i.e. memory is a concern) namedtuple might be a better choice.

One argument I've seen for using dictionaries is that namedtuples restrict you to column names that are valid Python identifiers, which is a smaller set than postgres allows for column names (postgres allows the '$' and non-Latin characters).

So does python:

>>> d = {'ñanáöß$': 1} >>> d {'ñanáöß$': 1} >>> d.keys() dict_keys(['ñanáöß$']) >>>

The parent was talking about namedtuple.

    >>> collections.namedtuple("test", "$")
    Traceback (most recent call last):
    ...
    ValueError: Type names and field names can only contain alphanumeric characters and underscores: '$'

That's because attribute names can't be called "$", so `sometuple.$` would be a syntax violation.

I guess everyone is aware of that, thanks,

The parent comments were discussing why choosing dicts over namedtuples for db-originated columns, and the fact that field names are restricted on namedtuples but not on dicts is a very compelling point.

Making examples usind dicts and explaining why namedtuples have restrictions is completely missing the point.

HN supports "<pre><code>" blocks in a similar manor to Markdown, but using a two-space prefix (rather than Markdown's 4-space prefix):

  >>> d = {'ñanáöß$': 1}
  d {'ñanáöß$': 1}
  >>> d.keys()
  dict_keys(['ñanáöß$'])

Named tuples seem to offer the best of both worlds, as they're immutable.

This memory waste argument is only true if you want to fetch all rows, which you could possibly avoid with different techniques. Otherwise, a dictionary seems like a very simple, convenient, Pythonic, and intuitive representation of a row.

This lib also looks brand new, so there is plenty of time for the community to optimize the hell out of it.

If this were a big deal, it could be optimized using something like a key-sharing dict implementation [1]. You can keep the dict interface to accommodate column names that aren't valid identifiers, and avoid most of the memory overhead.

[1] https://www.python.org/dev/peps/pep-0412/

python-sqlite3's RowFactory does this as well.

I've made a pull request implementing namedtuple support at https://github.com/kennethreitz/records/pull/6.

I'm also a fan of NamedTupleCursor. Not sure how NamedTuples compare to raw tuples in terms of memory consumption, but they make code look a lot nicer.

They're essentially the same footprint IIRC.

I also "voted" for NamedTuples https://twitter.com/nathancahill/status/696146948567359488

Namedtuple CPU-speed performance for some common operations is TERRIBLE compared to dictionaries, at least on 2.7. Orders of magnitude difference. So bad that it really matters. It's shocking how bad the implementors got this.

Curious what the operations you're talking about are. Get?

If you look at the the StackOverflow in my sibling comment to yours, running 'obj.attrname' on a namedtuple takes longer because it needs to translate 'attrname' to an integer index and then run (e.g.) 'obj[0]'. Outside of that, I'm not sure.

Pickling, for one.

  >>> nt = namedtuple('nt', 'a b')
  >>> nt10k = [nt(1, 2) for i in range(10000)]
  >>> dict10k = [{'a':1, 'b':2} for i in range(10000)]
  >>> timeit pickle.dumps(nt10k)
  10 loops, best of 3: 26.3 ms per loop
  >>> timeit pickle.dumps(dict10k)
  100 loops, best of 3: 2.49 ms per loop

namedtuple is just a thin wrapper around tuple. It literally constructs a class definition that sub-classes tuple as a string and then executes it. You can see the string template that it uses here[1]. If you're interested in something like namedtuple there are other[2] things[3] you can use depending on your use-case.

[1] https://github.com/python/cpython/blob/master/Lib/collection...

[2] http://stackoverflow.com/a/2648186

[3] https://pypi.python.org/pypi/frozendict/

It's really weird that immutable tuples take a performance hit, compared to mutable lists (almost 50% on access!?!).

That's because Python 2 has a special case for lists[1]. This does not exist in Python 3[2] (and therefore neither does the performance difference).

[1] https://github.com/python/cpython/blob/2.7/Python/ceval.c#L1...

[2] https://github.com/python/cpython/blob/master/Python/ceval.c...

It looks like list access got slower as a result though.

probably so you don't have to remember what order they were returned in

Reply on news.ycombinator.com