Hacker News Reader

Hacker News

Comment by _hyn3 | Hacker News Reader

Parent Full thread

_hyn3

This doesn't feel right.. it's not actually a nice side effect at all.

1. it's not in the spec

2. you shouldn't rely on it

3. python can't figure out if you're relying on it, so no error will be raised

4. subtle bugs are sure to be introduced by people who "know" this "feature" exists and use it.

Regardless of the cool implementation details, this post shouldn't advertise that "keywords become ordered" and "A nice "side effect" of compact dict is that the dictionary now preserves the insertion order".

Ergo... we built this awesome thing that you'd love to use but you can't. Don't use it or you'll be a bad programmer who doesn't read specs!

(Just use addict or OrderedDict.)

View on HN

For what it's worth, the JS ecosystem has tried for years to get users to not rely on key iteration order. It was always unspecified but no matter what, users still expected them to iterate in insertion order.

V8 fought against it for years. Look at this bug horror show:

https://bugs.chromium.org/p/v8/issues/detail?id=164

I think Python is doing the right thing. The only reason to not provide a deterministic iteration order for unsorted maps is to give more room for optimizers. But it seems like there is a clean implementation that is faster while also iterating over keys in insertion order.

Sure, maybe someone will come up with an even better optimization in the future that would break this, but at some point, you have to say, "OK, what we have is good enough, and it gives users a more predictable system."

If you are going to declare insertion order is non-deterministic, you really should make it non-deterministic by doing something like shuffling it. Otherwise, users will just inadvertently rely on whatever deterministic-but-unspecified-order the current implementation happens to provide.

For once, C++ got some usability right and created std::map and std::unordered_map which have explicit key ordering behaviour.

Now, back to waiting for my project to compile.....

If you think offering every variation of data structures is doing usability right, then C++ has done a lot of usability right. So much that its usability has suffered.

Java as well. Need sorted? TreeMap. High-performance but unordered? HashMap. Preserve insertion order? LinkedHashMap.

I would guess it probably is the same way in c++ but I've never spent more than 5 seconds considering which collection to use in Java. They have a decent amount of them but once you are aware of their existence they are incredibly intuitive

FWIW Python has had OrderedDict for years.

> make it non-deterministic

In Python v3.1 - v3.5 the order is deterministic within an execution, but non-deterministic across executions. This was implemented to avoid the possibility that a hacker could slow down dict key lookup/insertion by creating a large number of hash table collisions.

https://mail.python.org/pipermail/python-announce-list/2012-...

IMHO, that doesn't seem to be a big vulnerability.

It never seemed to catch on with the bad guys, but that doesn't mean it wouldn't have happened if major languages hadn't implemented counter measures.

DDoS using DNS amplification was possible for decades but never happened. Then one day a bad guy got bored, wrote a proof-of-concept, and the rest is history.

At least with respect to ordered kwargs, the spec is changing:

https://www.python.org/dev/peps/pep-0468/

I believe it's an implementation detail that in cpython 3.6, dict and odict are essentially equivalent now, but it's an implementation detail which is there to support an official feature of the language.

If your project can sustain a minimum version of (the as yet unreleased) Python 3.6, I don't see why you shouldn't begin to rely on this feature straightaway.

Ugh. This is all bad.

This just isn't the sort of thing that benefits from fucking with.

This is imposing a slight but not-insignificant open-ended technical debt on the whole of the Python ecosystem, for what seems to me to be very obscure benefits.

> obscure benefits

1. No need for metaclasses simply to make the class construction locals dict an ordered dict. ORM library devs will be happy.

2. No need to pass a list of tuples when you wanted to pass keyword arguments. That's a big win for usability.

> technical debt

The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt?

> This just isn't the sort of thing that benefits from [change].

The core devs say that status quo is the sane default. They're not just mucking about for the amusement of code churn.

> The language spec will say that class construction locals and kwargs are insertion-ordered, but other dicts are ordered according to the whims of the interpreter. What is causing technical debt?

The debt will accrue in the wider ecosystem. The spec doesn't guarantee dicts are ordered, but people will rely on it. Then, when a new dict implementation comes along, all code written based on a (then) correct assumption (but not based on the spec) suddenly stops working.

If that happens -- people begin to rely on dict order -- the core devs will consider that when deciding whether or not to change the implementation.

Or they could just flush this now and save all the trouble.

Dicts with integer keys have always had a very predictable order. Do you consider that technical debt?

Only if someone writes (and shares widely, to be used) code that depends on it.

I've [ab]used the stability of dict order when you haven't modified the dict in the meantime, in one-off scripts.

But I wouldn't write code that depended on predicting dict key order though, that seems a bit too much to me.

----

For me, I feel that a Python syntax text should run unmodified on as many interpreters as possible, cPython, Pypy, Jython, Brython et. al., embedded, etc. There are lots of Python and Pythonish runtimes out there, beyond just what core dev produces.

If I can write code that works on most of those systems out-of-the-box that's a huge win from my point of view.

It should be clear that changing the semantics of the interpreter, even in subtle ways, puts a burden onto anyone hoping to maintain parity in their own runtime, yes?

> It should be clear that changing the [behavior] of the interpreter, even in subtle ways, puts a burden onto anyone hoping to maintain parity in their own runtime, yes?

No, that's not clear. Any changes to the language specification put a burden on the maintainers of interpreters/compilers. Any extra changes to the CPython interpreter do not. You argued that behavior of CPython becomes its own spec, beyond that of Python itself, but this does not seem to be the case. There are a bunch of optimizations that CPython does, like caching small ints, that programmers do not tend to rely on and do not form a sort of shadow-spec for other interpreters.

You might say that dict keys staying in insertion order is more useful than the other CPython idiosyncrasies and therefore will become a shadow-spec, unlike the others. That's possible, but let's consider the risk.

Pythonistas have done just fine without this behavior for a while. There's OrderedDict, but it was added in 2008, relatively late. If there had been lots of demand/usage, I'd have expected it to be added earlier. PEP 372 [0] indicates that use in a metaclass was one of the big motivations, which was only enabled the year before.

When a new Pythonista considers this issue, they are likely to search the internet for discussion and will find commentary on OrderedDict. I expect most will use OrderedDict when appropriate.

The riskiest group would be folks porting code from PHP and Ruby, or other languages that have a core mapping type that keeps insertion order according to the language spec. They'd just expect a Python dict to behave the same way and won't bother to read about it. To evaluate the impact of this change, we should ask how much porting from PHP/Ruby/etc. do we expect to occur, how likely is it that dict implementation will drop insertion ordering in the future, and how much benefit does this implementation provide.

Based on the explanation of the memory and compute savings, this new dict implementation sounds like a good idea.

Also, Ruby made the change in 2009 to the language spec. I'm not an expert Rubyist, but AFAICT it hasn't caused any problems for their community.

[0] https://www.python.org/dev/peps/pep-0372/

> No need for metaclasses simply to make the class construction locals dict an ordered dict. ORM library devs will be happy.

understatement with that. Saw someone trying to avoid using a metaclass for that, "had" to use descriptor objects which all used the same global counter to work out their order

Hey, guess what? Just don't use metaclasses. At all, for anything. Done.

GvR called it "The Killing Joke" for a reason.

This is a clear case of "negative productivity": People are working "harder than a cat trying to bury a turd on a marble floor" to pee in my soup. It makes me grumpy.

what, not using metaclasses in that case caused more "issues" than would of been caused by using metaclasses

multiple global values to track state, since can't bind the state to the class' creation, since the conversion to the final form of the class happened after its creation, vs being able to bind the state to the class' creation, since the conversion happens as the class is created

Knowing nothing more about this codebase than the above, I would fire everyone and make the CEO's nephew write it. It could hardly turn out worse and it would be a lot cheaper.

(I'm joking, but only barely.)

The reduced usecase (story?) is the following

The customer requests the following

    @?
    class somename (?):
        an_attr_a = ADescriptor(somestuff)
        an_attr_1 = ADescriptor(somestuff)
        an_attr_a1 = ADescriptor(somestuff)

One of their expectations is that the attributes have their definition order stored somewhere. You may either use the decorator method, or the metaclass method

I don't get your concern. You can always treat an ordered collection as unordered without harm, aside from maybe not picking the most optimized solution had you considered order. Existing code will assume it's unordered, code written for 3.6+ can assume it's ordered.

> assume it's ordered

That's discouraged for general dict usage, as non-CPython interpreters may choose a different implementation. Only kwargs and class creation dicts will be specified as ordered.

Absolutely, I just meant in the specific cases where it's guaranteed by the updated spec.

I'm imagining a scenario where I want to back-port Python 3 code to work with Python 2 and now must take account of it relying on ordering (that's not guaranteed by Python 2 semantics.)

Also it's just bad design. Ordering is an additional constraint so I feel that it should require additional code ("OrderedDict" vs "dict"). It's a deep semantic change for what seems to me to be bullshit superficial justification.

I tend to agree that this is something that could be confusing to rely on, especially if running tests against multiple version of Python (such as a previous version where this isn't the behavior).

    Explicit is better than implicit.

    There should be one-- and preferably only one --obvious way to do it.

In python 2 the ordering was deterministic. Cue lots of weird bugs when upgrading to python 3. Of course, it should never have been relied on... but it had been.

I'd prefer either a completely random order or a completely deterministic one - but please no take backs (people will rely on it).

I'm not sure anybody should be relying on code that relies upon the order of an unordered dict.

These reliances usually aren't explicit, they just happen to work that way and it can be very hard to tease out the dependency unless you have a way to easily screw with iteration order.

Pretty much any unspecified but (because of implementation) reliable ordering will eventually end up with code relying on it, usually implicitly, e.g. some Go code broke in 1.5 (and the release notes noted it explicitly) when scheduling order changed[0] despite it always having been undefined. In fact, specifically to catch this kind of hidden dependencies the Go developers added limited scheduling randomisation to the race detection mode.

[0] from definition-ordering to last-definition-bias e.g. if you launched 3 goroutines 1, 2 and 3, in 1.4 the scheduler would just run them in that order, in 1.5 it would run 3 first followed by 1 and 2.

That's true, but to rely on these behaviors is to introduce bugs into your code, and we should be detecting and correcting them, not forgoing semantically clean optimizations because someone's buggy code will break.

And so we get a 3rd way to interpolate strings in 3.6.

Yeah, this is not a dict that has ordered keys. It's a dict that has that side effect.

I can't think of any reason why a set's members, or a dict's keys (which are just set members with associated values, conceptually) should be unordered. Yes, set membership is traditionally unordered, but we have ordered sets and dicts, so they must be desirable, and they're certainly coherent.

I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions.

> I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions.

Unspecified-order minimizes constraints on future implementations. If someone finds a useful time or space optimization (for particular workloads, not necessarily in big-O asymptotic terms) that isn't consistent with the specified ordering, that might be a compelling reason to implement a version without the ordering constraint. (But, if its not the best choice for general use, that's true even if the default implementation isn't ordered, since its possible that the optimization will still be unsuitable as the default.)

And, obviously, even if the base case has one ordering, there's all kinds of reasons you might need to implement a version with a different ordering.

"Ordered sets" are coherent in the sense that you can make a data structure like a set and then define an order over its members, and call it an ordered set, but "ordered sets" are not sets.

There are certain purposes sets are natural for. When you introduce ordering, you will naturally be asked to add methods for querying or perhaps modifying that ordering in various ways. Many methods later, it's worth asking why you are calling this data structure a set, and what its actual purpose is supposed to be.

I interpret it differently: the ordering is a side effect, and the positive aspect is that it's easier to debug conveniently, not a feature you should rely on version to version.

I am not very sure this is tat bad. If code breaks because dict is no longer ordered, people should have used an ordered one from the start. It's a happy accident it is faster and ordered and not bad because whoever relies on dicts bein unordered (when sometimes they aren't) deserves whatever breakage they get.

This is very similar issue that made migration to Python 3 so painful.

A lot of the code had broken unicode from the start. The difference was that it worked in python 2 most of the time and failed on special cases, while python 3 was stricter and mixing bytes with string always caused it to crash.

Once dict() is ordered and people will use that property (even unknowingly) if the behavior changes again in the future, a lot of their code will be broken.

It may be similar in that it allows for relying on consistent behavior when one shouldn't, but fixing it seems a lot easier than decades of not thinking about Unicode.

A language that makes it easier to write shitty code is worse than one that doesn't

I thought Lisp was the best language though, you're telling me it's the worst? :-p

It isn't right, but popular personalities like the creator of Flask have been demanding it for years so here we are. Welcome to Open Source.

Reply on news.ycombinator.com