Hacker News Reader

Comment by Too | Hacker News Reader

It seems like the python maintainers always try to complicate things for themselves to cater for some obscure use case that nobody use. I mean really, when was the last time you ever accessed a namedtuple by index?? (For vectors i do see a benefit that you can throw it right into a matrix multiplication that is designed for arrays but really, you can do that equally easy in other ways). And constructing them on the fly doesn't make much sense either as it defats the benefits of a shared type across the code base.

All people ever wanted was just a way to write in one line ´Point3D = struct(x, y, z)´ that translates into the exact equivalent of:

    class Point3D: 
        def __init__(x, y, z):
            self.x = x
            self.y = y
            self.z = z

What would be the startup performance of parsing the above 5 line class compared to a namedtuple? Surely it should be faster to create it with a shorthand form built into the language, if the functionality is equivalent?

View on HN

One advantage of namedtuple is that it is just a tuple! You can pass the namedtuple to code that expects a tuple, even C code that expects a tuple. This makes it easy to add names to tuples after the fact without breaking existing code.

In one project, I use a Point2D class as a namedtuple. Because it is just a tuple, I can easy convert it to a numpy array when working with a collection of points.

> I mean really, when was the last time you ever accessed a namedtuple by index?

Yesterday. I use this quite a bit. I like non-named tuples as a data structure and use them regularly, and often partially upgrade a tuple to a named tuple.

Changing these sorts of things is what caused the Python 2/3 split, I don't think we as a community want that again for a while, so backwards compatibility for things like this that aren't _that_ uncommon should be a priority.

"Partially upgrade a tuple"? What does that mean?

It means that if you have a data structure that is represented by a tuple, you can switch the creation call with a namedtuple of the same size, keeping the tuple API, therefore not breaking any older code that may rely on it, while at the same time providing a cleaner interface (using the attribute names) to newer code.

Of course, old code should be upgraded to use attribute names, but it's really convenient to be able to upgrade it painlessy.

As a concrete example, in Python 1.5.2, urlparse.py:urlparse returned a 6-element tuple:

  # Parse a URL into 6 components:
  # <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
  # Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
  # Note that we don't break the components up in smaller bits
  # (e.g. netloc is a single string) and we don't expand % escapes.
  def urlparse(url, scheme = '', allow_fragments = 1):
      ...
      tuple = scheme, netloc, url, params, query, fragment
      _parse_cache[key] = tuple
      return tuple

This API was around for years, with people using tuple indexing either explicitly or through destructuring assignment.

This was replaced with a namedtuple. The following is from Python 2.7.10:

  class ParseResult(namedtuple('ParseResult', 'scheme netloc path params query fragment'), ResultMixin):

    __slots__ = ()

    def geturl(self):
        return urlunparse(self)


  def urlparse(url, scheme='', allow_fragments=True):
    """Parse a URL into 6 components:
    <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
    Return a 6-tuple: (scheme, netloc, path, params, query, fragment).
    Note that we don't break the components up in smaller bits
    (e.g. netloc is a single string) and we don't expand % escapes."""
    tuple = urlsplit(url, scheme, allow_fragments)
    scheme, netloc, url, query, fragment = tuple
    if scheme in uses_params and ';' in url:
        url, params = _splitparams(url)
    else:
        params = ''
    return ParseResult(scheme, netloc, url, params, query, fragment)

The old tuple-based API still works, so it doesn't break backwards compatibility, while it also gives people the ability to migrate to an attribute-based API.

That said, it's very hard to remove the old API. The latest version of Python, in url/parse.py:urlparse, still supports tuple-based indexing.

Perhaps Python 4 will remove that behavior? Probably not. At the very least there would need to be a namedtuplemigration type which gives a warning if the old tuple API is used when only attribute lookup is expected.

> This API was around for years, with people using tuple indexing either explicitly or through destructuring assignment.

FWIW destructuring assignment uses the iteration protocol[0] not indexing.

[0] which can fallback on indexing if there is no explicit iteration support, IME that tends to be more unhelpful than convenient

Hmm, I think meant to say something like "using either explicit tuple indexing or through destructuring assignment". Words, mixed up, they were. But yes, what I wrote was wrong.

Thanks for the great answer!

Agreed, I use the fact that "namedtuples are tuples" constantly. For one thing, you can pass them to 3rd party libraries that take tuples while maintaining readability on your end. I would be horrified if they dropped index access. It would break everything.

I should say though, for simple POJO-style classes @attr.s (https://github.com/python-attrs/attrs) has replaced 90% of my namedtuple usage.

> All people ever wanted was just a way to write in one line ´Point3D = struct(x, y, z)´ that translates into the exact equivalent of [... a regular class with just an x, y, and z attributes].

You're giving up a lot:

* The namedtuple is dramatically more space efficient (which will be important if you have a lot of points).

* The namedtuple has a nice self-describing __repr__() (which will can important when you're debugging):

   >>> focal_point
   Point3D(x=10.2, y=8.5, z=13.1)

* The namedtuple defines an equality relation (which is important if you ever want to recognize that two points are the same or need to find-out whether a given point is in a container):

    >>> new_point in visited_points
    True

* The namedtuple is hashable (this will be important if want to pass points into a function that uses functools.lru_cache() for example).

* The namedtuple is unpackable. This will important with loops like such as:

  for x, y, z in points:
     print(x*y - y*z)

* The namedtuple works with just about every existing tool in the language that expects regular tuples (such as old-style string formatting).

    >>> 'Horizontal: %f  Vertical: %f  Depth: %f' % focal_point
    'Horizontal: 10.200000  Vertical: 8.500000  Depth: 13.100000'

* The namedtuple has lexicographic ordering (which may be nice if you want to pprint() a set of points in a deterministic order).

* The nametuple has a _make() method for casting existing tuples without unpacking them . This is useful with the sqlite3 and csv modules:

  EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade')

  import csv
  for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
      print(emp.name, emp.title)

  import sqlite3
  conn = sqlite3.connect('/companydata')
  cursor = conn.cursor()
  cursor.execute('SELECT name, age, title, department, paygrade FROM employees')
  for emp in map(EmployeeRecord._make, cursor.fetchall()):
      print(emp.name, emp.title)

* The namedtuple is sliceable (useful for computing projections or partials to 2D points):

  >>> focal_point[:2]
  (10.2, 8.5)

* Many other little things as well: conversion to a dictionary with _asdict(), cloning with "newpoint = oldpoint._replace(y=0, z=1)", etc.

Maybe exact equivalent was an exaggeration, but anyway..

Half of those examples are just valid because x,y,z in particular have a de-facto ordering that everybody follows, for any other record type it's just confusing (what does slicing an EmployeeRecord mean??!). Csv module will produce named records based on the header so i don't see the use there. With sqllite3 you can set row_factory=Row which will produce results which already has access by name. If you for some reason need to convert to a dictionary, you could always use __dict__

Comparability, space, hashing and immutability are important though. Altough frankly, i can't take any immutability argument seriously since python has refused to add a constants for decades, it's just hypocrisy to one day have the philosophy that everything is dynamic and then then the next day say it's not.

I agree. Change the class slightly and several of those methods become a big negative. As I've argued, I don't want to include API features "for free" if I don't really need them, because they will become a support headache for me in the future

Here's what I mean. Let's start with the Point(x, y, z) class. If I want to tweak it a bit to add a pre-computed 'r' attribute, I can do the following in a normal class:

   class Point:
      __slots__ = ("x", "y", "z", "r")
      def __init__(self, x, y, z):
        self.x, self.y, self.z = x, y, z
        self.r = (x*x+y*y+z*z)**0.5

(If I want to enforce immutability I would use the attrs library.)

I can't simply tweak a namedtuple to support that, because the "r" will leak in to the __init__, _as_dict(), __iter__, __repr__, and other methods. Nor can I write "for x, y, z in points" any more.

Now, consider a point expressed in spherical coordinates (r, theta, phi). The two points (0, 0, 0) and (0, 45, 180) are identical, because both are at the origin. The two points (10, 0, 0) and (10, 360, 0) are also identical, because both are at (10, 0, 0) in Cartesian coordinates. (Angles expressed in degrees.) Either the __init__ needs to normalize the values to some standard range, or the __eq__ needs to check for degenerate cases. The default __eq__ and __hash__ for namedtuple are valid if all coordinates are in standard form, but that's incomplete.

(Huh, and I see that the tuple behavior is that (nan,) == (nan,) even though nan != nan. This doesn't makes sense for some cases.)

I disagree with the proposal that slicing is "useful for computing projections or partials to 2D points". In addition to the weirdness in slicing an Employee record, how do you project along the y axis using slicing? Are people really supposed to write [0::2]? Or [2:-1:-2] if they want the projection on the other side? That seems like a horrible API. Plus, the result is an anonymous tuple, when a Point2D makes more sense.

For that matter, in spherical coordinates there is no meaning to point[::-1], so why do I want the API to support that in the first place?

Next, consider a point expressed in homogeneous coordinates x, y, z, w. Slicing here is worthless as a way to give a projection. What does point4[2:] mean?

In a real system of course, you project given a view point and orientation, and are not limited to a projection along the axis, which is all that a slice might be able to give you. Of course, in real system you store all of the points in an array because that "is dramatically more space efficient".

Then there's the _make() method, which I'm struggling to understand why it's useful. It seems to save only a few characters over either of the following:

  for emp in (EmployeeRecord(*x) for x in csv.reader(file)):
      print(emp.name, emp.title)

  for row in csv.reader(file):
    emp = EmployeeRecord(*row)
    print(emp.name, emp.title)

The latter is also, IMO, easier to understand and maintain for a wider number of people.

On a deeper level, I can't see using this because most code would expect EmployeeRecord.age to be an integer. It's not easy to change the example code to, say, list those which are legally allowed to sell alcohol in the US:

    for emp in map(EmployeeRecord._make, csv.reader(open("employees.csv", "rb"))):
      if emp.age >= 18:
        print(emp.name, emp.title)

Even changing it to "18" gives a problem because "100" < "18".

At least the sqlite example will (likely) put the right types in the object, but again it's only saving a couple of keystrokes at most. Why learn a specialized API function for that?

As for the "Many other little things", those appear to be solutions in search of a problem. As Too points out, if I needed an _asdict() I could often do obj.__dict__.copy(). In practice, I use __dict__ access in debugging - rarely - and don't recall ever using that in production code. Why is _asdict() so important that it should be part of my API? If it is so important, why don't more people include it with their own class definitions?

I don't want to litter my API with functions that are useless, which is what namedtuple invites me to do by having me pretend that my data is "super" tuple-like.

>It seems like the python maintainers always try to complicate things for themselves to cater for some obscure use case that nobody use. I mean really, when was the last time you ever accessed a namedtuple by index??

I do things like

    for x, y in iterable_of_ntuples:
        #process

all the time

That can be solved by implementing __iter__ method on the class, isn't it?

No, it's an iterable containing named tuples, not iterating over a named tuple.

I suppose if python had object destructuring like newish javascript (I think that's what it's called) it would be trivial. I don't know what you'd do instead in this case that doesn't involve adding additional methods or something.

Object destructuring is tuple unpacking in python which is exactly what the code example does. What the parent was remarking on was caring about the performance of that rather than the ability.

Except in javascript you can unpack by name. This would solve this issue as well.

That's not the point, destructuring works fine on classes implementing `__iter__`:

    class A:
        def __iter__(self):
            return iter(range(3))

    for x, y, z in [A(), A(), A()]:
        print(x, y, z)

Prints

    0 1 2
    0 1 2
    0 1 2

TIL, cheers!

It's both. The unpacking of the individual tuples are managed through the iterator protocol as well.

Sure, but if you want an immutable object with iter, reasonable equality operators baked in, and named access, you can either define iter, eq, neq, and le on a new style slotted object with a bunch of getter descriptors, or you can make a namedtuple.

Your version is mutable, whereas a namedtuple is immutable. a very important difference for my field. I use namedtuples all the time.

Also, I can't expand an instance of your Point3d as arguments to a function, DrawPoint(*p), I would have to write DrawPoint(p.x, p.y, p.z), did I preserve order correctly?

Whereas a namedtuple expands and preserves order.

Wouldn't it be so convenient if Python provided a way to define immutable classes? Also, you can always provide a `draw_point(*p.args)` which isn't bad at all.

You can get close. Set __slots__ and then define each as a property with no setter and no deleter. However, an object can still be modified if it stores a mutable object (list, dict) and, while I'm not in front of a computer atm,I believe you can overwrite __slots__ to do whatever.

Here's an SO answer that explains more than I'll ever know about slots. https://stackoverflow.com/questions/472000/usage-of-slots

Re: storing mutable objects, yea but tuples do that too:

    t = ({}, 1)
    t[0]['foo'] = 'bar'

Your Point3D class is _not_ functionally equivalent to a NamedTuple. The class is so minimal that I wouldn't bother calling it a class as much as I would call it a glorified dictionary.

I do like the concept of Structs. I first used them in Ruby, so I figured that all "very high" languages had the concept. Python, however "struct" has a very different meaning in Python. sigh It's intended for encoding/decoding binary data from strings/bytes.

https://docs.python.org/3.6/library/struct.html

There is a lesser known use case for the type(...) function. Normally, one would use it to find out an object's type.

     >>> type(42)
     <type 'int'>
     >>> type("foo")
     <type 'str'>
     >>> type(type)
     <type 'type'>

But, it can also be used to create new class types. https://docs.python.org/3.6/library/functions.html#type

     >>> Point3D = type("Point3D", (object,), dict(x=0,y=0,z=0))
     >>> my_point = Point3D()
     >>> my_point.x
     0
     >>> my_point.y = 42
     >>> my_point.y
     42
     >>> my_point.a
     Traceback (most recent call last):
       File "<stdin>", line 1, in <module>
     AttributeError: 'Point3D' object has no attribute 'a'

Which is, essentially, what Ruby's Struct class provides. And, Python objects don't have privacy, so it's more comparable to the OpenStruct class.

Edit: Just wanted to tack on a shout out to Attrs which is really nice improvement to "Struct" concept. http://www.attrs.org/en/stable/ h/t hynek

>Python, however "struct" has a very different meaning in Python. sigh It's intended for encoding/decoding binary data from strings/bytes.

That's because the struct module is for dealing with structs, in the C/C++ sense. Like actual structs.

That aside, attrs is a great library!

A good link about metaclasses in Python:

https://stackoverflow.com/questions/100003/what-is-a-metacla...

Namedtuple is used a lot in the standard library to return struct-like results which used to be returned as an actual tuple, so making them non-indexable would break a lot of code, some of which was written before namedtuple even existed.

I did that analysis a day ago when this topic came up previously, at https://news.ycombinator.com/item?id=15136961 .

There are 13 locations in the standard library where a namedtuple was used to replace a tuple, where making them non-indexable would break code.

There are 8 locations where it's used as a shorter way to write a simple class. My point in that thread was that namedtuple should never be used this way.

I think namedtuple is a terrible hack that never should have been (at least not this way), but changing it is going to be a PITA.

Notice how when someone asks something like that, some bizarro practitioners of such methods would always appear and say how they are invaluable for them...

It's invaluable in several use cases, especially when working with external code that expects a tuple, when unpacking a tuple (a, b = c) especially when iterating over them in a loop (for a, b in c).

You have used some of these methods before. So are you bizzaro?

No. Nobody is bizarro for mentioning this, it's extremely handy and it's kind of in the name: namedtuple. Without this they are utterly pointless.

It's called a tuple so why would accessing its elements by index be "bizarre" in any way?

Because order should be unimportant for a tuple.

Are you joking or serious? I can't read your tone without emoji.

Serious; see coldtea's reply for a fuller explanation.

Or consider (firstname="John", lastname="Doe") vs (lastname="Doe", firstname="John").

I think you'd agree that they're the same.

They are the "same" by equality, but that's not the only relevant comparison. Ordering is important if you want lexicographic less than.

Without ordering, you wouldn't be able to sanely run `sorted` on a iterable of namedtuples.

Dumb question from a non-developer: wouldn't using a dict object make more sense if someone wanted an unordered named tuple?

It's not about the storage (besides Python also has order dicts).

Namedtuples are for the convenience of creating a record type (though indeed as others say, treating it as tuple where things expect a tuple is also a benefit).

A dict is appropriate for labeled data. Currently unordered, but likely to keep insertion order in v3.7 and later.

Don't call it an unordered namedtuple. That's confusing, because tuples are ordered.

You could order with a custom comparison; specify the columns* to be ordered as in a SQL 'ORDER BY'.

I would expect that construction to result in identical behavior for dict() and certainly not anything calling itself a tuple. NamedTuples behaving in the manner you expect violates LSP with respect to tuple().

I disagree. As of v3.6, the order of keyword arguments is preserved.

Typically tuples are ordered lists of values. See Codd's relational model, Linda, Erlang, etc.

In the relational model the order shouldn't matter:

  A relation is defined as a set of n-tuples. In both mathematics and the 
  relational database model, a set is an unordered collection of unique, non-
  duplicated items, although some DBMSs impose an order to their data. In 
  mathematics, a tuple has an order, and allows for duplication. E. F. Codd 
  originally defined tuples using this mathematical definition. 

  Later, it was one of E. F. Codd's great insights that using attribute names 
  instead of an  ordering would be so much more convenient (in general) in a 
  computer language based on relations. This insight is still being used today. 
  Though the concept has changed, the name "tuple" has not.

I stand corrected; I was probably fooled by the naming. Still, in most contexts tuples are ordered while sets/bags are not.

Order is important for "pure" Python tuples, so I see no reason why it shouldn't be the case for namedtuple. Mathematical tuples are ordered as well:

"In mathematics a tuple is a finite ordered list (sequence) of elements. An n-tuple is a sequence (or ordered list) of n elements, where n is a non-negative integer. There is only one 0-tuple, an empty sequence. An n-tuple is defined inductively using the construction of an ordered pair." (Wikipedia)

Tuples are ordered by definition. We use names for things to convey what they are, not what we want them to be. If you want a construct in which order does not matter, call it a set or a dict.

Followup:

Should order be important for a Tuple under a system that defines a Tuple as an immutable ordering?

That is, whatever your philosophical issues with namedtuple ordering, in python, a tuple is an immutable ordered list of things. In practice, tuples have taken the role of being type-heterogenous, known length, records.

One might then reasonably expect a namedtuple to be a type-heterogneous, known length, record, where attributes are named.

FWIW I was referring to the abstract pure tuple, not the tuple as implemented in Python and some other languages where order is very useful.

"inessential" might be a better than "unimportant"

Because the whole point of namedtuples is to have them as a convenient quick notation for records.

But I was referring to the parent's general observation that about "catering to some obscure use case that nobody uses", which I agree with -- not about their specific example.

some bizarro practitioners

That would be the bizarro practitioners who expect subclassing to actually be subclassing. Maybe there is a sane argument that namedtuple shouldn't be a subclass of tuple but that ship sailed a long time ago.

Yes, it should be a special construct (record).

Python has really painted themselves into a corner. The language is so extremely dynamic with everything being allowed and people actually relying on such things that you can't even look at the language without breaking backwards compatibility. https://xkcd.com/1172/ is a very accurate description of the current state.

Python has painted itself into the top position on the most popular list,and has been in the top three for as long as I can remember, mostly because of its inherent internal dynamism. People relying on decades old language features is the opposite of a problem.

https://www.extremetech.com/computing/252987-python-tops-lis...

People use python because it has nice syntax, batteries included standard library, easy to make code for both windows and Linux, no compilation, big community, and it's the closest arm to reach out to when you realise your shell scripts are getting too big.

Not because you can monkeypatch the _attrs_ function on the fly somewhere on a class in the standard library.

I'm flattered :)

Although I can't tell if the moral of this is "if the api allows, it will be done, and you can't break the api later", or if the moral is "just because you think it's useless doesn't mean it really is".

While I don't explicitly use index access to NamedTuples, I do make use of NamedTuples being a subclass of tuple, and I know I'm not the only one.

> when was the last time you ever accessed a namedtuple by index

If you don't want that, why are you using "Named[ThingWhichWasOnlyAccessibleByIndex]"

Use this instead, or something

    class Struct(object):
        def __init__(self, **kwargs):
            self.__dict__.update(kwargs)

> What would be the startup performance of parsing the above 5 line class compared to a namedtuple?

I think https://news.ycombinator.com/item?id=15135147 explains why you can't optimize namedtuple like that, since dynamic dispatch makes calls to them not "the exact equivalent" of that code. The proposal to move them to a top level data structure could solve this problem though, I think.

Checkout [attrs](http://attrs.readthedocs.io/en/stable/) for a very powerful, yet equivalently short way of defining classes. Personally, I prefer defining them with the decorator rather than the shorthand, but attrs is definitely my library of choice for value semantic types.

> I mean really, when was the last time you ever accessed a namedtuple by index?

They are good for database stuff. The row gets stuffed into the tuple, by index, and then you get to write readable code accessing the fields by name.

A tuple is immutable, your `Point3D` is not. It's a huge difference.

The main issue I see is caring too much about other implementation of Python.

While that is a very kind behavior, it brings more complexity to the issue.

The CPython developers should, imho, define/consolidate the API and then choose the implementation that yields the best performance for their implementation.

Developers of other Python implementation will surely find a way to implement that API with optimal performances with regards to the peculiarities of their own implementations.

Reply on news.ycombinator.com