Hacker News Reader

Comment by Pinus | Hacker News Reader

It's probably possible to do this in a one-liner using half of the stuff in itertools, but just changing the output data structure to a dict (at least temporarily; one can always make a list from it at the end) would yield code that is short, understandable and gets the job done without linear searches.

Essentially, I'm agreeing with eska's comment, with the addition: "...but consider your data structures!"

groupby looks neat, but it requires sorted input, so it's very easy to convert an O(n) problem to an elegant but O(n log n) program.

EDIT: one could also do it with a one-liner in pandas, which means that as long as the input is less than ten million lines or so, the "import pandas as pd" statement is going to take longer than the actual program...

View on HN

Perhaps not one line, maybe not perfect Python but definitely O(n):

    groupby_dict = collections.defaultdict(int)
    for l in mylist:
        groupby_dict[l['thing']] += l['count']
    newlist = []
    for thing, count in groupby_dict.items():
        newlist.append(dict(thing=thing, count=count))

Knowing how do that does require the deep knowledge of data structures I'd expect every professional programmer to have, and doing a quick scan of the languages standard library (maybe an hours work) to see what it offers. After that you're set for solving not only this problem, but almost guaranteed to solve all problems you are likely to hit pretty optimally in Python. I have no idea where Stack Overflow fits into the picture.

If you go a step further and do the Python standard tutorial found in the docs, you would discover how a finger weary experienced Python programmer might write those last three lines:

    newlist = [dict(thing=thing, count=count) for thing, count in groupby_dict.items()]

Comprehensions are sweet Python syntactic sugar, but unlike mapping your existing encyclopaedic knowledge of data structures onto the languages standard library they aren't necessary for a casual user of the language. Indeed, some Python'istas will tell you resisting such delights make for clearer code.

But trusting a quick Stack Overflow to tell you the optimal way to use a languages data structures - you must be kidding me.

Doesn't detract from any of your points, just as a FYI: Python has a counter data structure in the standard library. The "pythonic" way would be something like this:

    groupby_dict = Counter()
    for r in response:
        groupby_dict.update({r['thing']: r['count']})

Counter is a subclass of dict, no further conversion needed.

Deep knowledge is overstating it I think. Knowing a few data structures and their Big O notation for various operations goes a long way. You don’t need to exactly know how it is implemented under the covers. SQL is similar that once you grasp what data structure it uses to store your data, you can intuit what should be fast and what is slow.

Reply on news.ycombinator.com