Python 3.6 dict becomes compact and keywords become ordered

In python 2 the ordering was deterministic. Cue lots of weird bugs when upgrading to python 3. Of course, it should never have been relied on... but it had been.

I'd prefer either a completely random order or a completely deterministic one - but please no take backs (people will rely on it).

And so we get a 3rd way to interpolate strings in 3.6.

Ugh. This is all bad.

This just isn't the sort of thing that benefits from fucking with.

This is imposing a slight but not-insignificant open-ended technical debt on the whole of the Python ecosystem, for what seems to me to be very obscure benefits.

> make it non-deterministic

In Python v3.1 - v3.5 the order is deterministic within an execution, but non-deterministic across executions. This was implemented to avoid the possibility that a hacker could slow down dict key lookup/insertion by creating a large number of hash table collisions.

https://mail.python.org/pipermail/python-announce-list/2012-...

IMHO, that doesn't seem to be a big vulnerability.

For once, C++ got some usability right and created std::map and std::unordered_map which have explicit key ordering behaviour.

Now, back to waiting for my project to compile.....

> I just hope that if sets and dicts become ordered by spec, that some day someone doesn't feel compelled to implement unordered versions.

Unspecified-order minimizes constraints on future implementations. If someone finds a useful time or space optimization (for particular workloads, not necessarily in big-O asymptotic terms) that isn't consistent with the specified ordering, that might be a compelling reason to implement a version without the ordering constraint. (But, if its not the best choice for general use, that's true even if the default implementation isn't ordered, since its possible that the optimization will still be unsuitable as the default.)

And, obviously, even if the base case has one ordering, there's all kinds of reasons you might need to implement a version with a different ordering.

"Ordered sets" are coherent in the sense that you can make a data structure like a set and then define an order over its members, and call it an ordered set, but "ordered sets" are not sets.

There are certain purposes sets are natural for. When you introduce ordering, you will naturally be asked to add methods for querying or perhaps modifying that ordering in various ways. Many methods later, it's worth asking why you are calling this data structure a set, and what its actual purpose is supposed to be.

A language that makes it easier to write shitty code is worse than one that doesn't

This is very similar issue that made migration to Python 3 so painful.

A lot of the code had broken unicode from the start. The difference was that it worked in python 2 most of the time and failed on special cases, while python 3 was stricter and mixing bytes with string always caused it to crash.

Once dict() is ordered and people will use that property (even unknowingly) if the behavior changes again in the future, a lot of their code will be broken.

I'm one of those weirdos who wants to continue to use Python 2 for the foreseeable future, so to me this is just introducing a way to break backwards compatibility all the way back to 2 for the benefit of a similar but ever-more-incompatible language that I'll never use.

I'm not trying to be negative, I just want to let you know that people like me are out there.

this is fine, but with PYTHONHASHSEED no longer impacting dictionary ordering, we need a new way to test our applications to ensure they aren't relying on dictionary ordering, without having to replace all dict / {} instances with "myapplication.utils.patchable_dict". What solution has been arrived at for this use case? (there IS a solution, right?)

Many, many people won't read the spec. And their code will break when the implementation changes.

> If none of the training material says or implies that the keys will always be ordered, why would a programmer?

Because people make mistakes.

Given how much Python I learned just by opening the interpreter and seeing what happened, I can definitely go along with the idea that this is bad.

> why would a programmer

How many programmers have you worked with? (People will assume all sorts of things that are not true if it makes their job slightly easier, or even harder)

why would a programmer? Because the programmer would just run the code and check what different methods return as result. Then if it looks like the ordered keys, said programmer will rely on this behavior.

For instance, I'm writing some javascript recently. I try a little array in the console, if Array.keys() returns the orders keys, I will assume it is, and would check the docs (where?) only if I had a problem. Something unstable should not look stable. I agree with the Golang way, make it look like what it is by the specs.

> a programmer should not assume the ordering of the keys

The Python community has many inexperienced programmers, even non-technical people. Particularly, it's so beginner friendly by making it very, very easy to get started with the REPL, etc. A lot of these programmers aren't even going to look at the docs and even if they do, might not understand them.

It is very easy to inadvertently add dependencies on key order. For example, if you are generating HTML, you might have something like:

    print("<a " + " ".join(k+"=\""+v+"\"" for (k,v) in d.items()) + "/>")

And d would be a dict of attributes for the anchor element. The code will produce different output depending on computer which is not what you want because it can cause subtle problems for other systems expecting a certain output.

Several compiler bugs have been of this kind so it is not only newbie programmers that make these mistakes.

Ordered kwargs is in a PEP[0] that is in "Final" status, meaning it's been accepted and implemented and is making its way into the language proper.

[0]: https://www.python.org/dev/peps/pep-0468/

EDIT: On rereading, I see that the parent and GP aren't specifically talking about the kwarg-ordering benefit, but the general dict-ordering benefit. I would be surprised if dict did not become a synonym for OrderedDict in the spec in the near future.

The only potential disadvantage of dict==odict is performance, as anyone relying on the current arbitrarily-ordered dict behavior by definition isn't worried about the underlying implementation moving to a deterministic-by-insertion-order dictionary.

I've seen some python 2 code "made use" of the deterministic ordering of dict. Migrating to python 3 sucked, it took a while to realize this was the reason for some failing tests.

It seems strange that that's what you'd be worried about. Your users found bugs in their own code. That's GOOD, isn't it? Does it matter that they blame you?

Like if I'm using pointers after freeing them and just happen to get usable data back... then you go an change it so that memory is reallocated and my code breaks. Yeah, it is a bug. In my code. It's pretty clear that I should not have been doing that. If I'm to stubborn or stupid to figure that out, then I'm really just a shit programmer.

chance my friend is what hides many bugs

Parent said for bugs happening in the other direction -- written for 3.6, run on earlier versions, so it would be the inverse happening (ordered assumption not holding).

Really? It's in the freaking name. Code is written that depends on dictionary being ordered. Works in CPython 3.6, fails on earlier CPython and other implementations of Python.

I suspect that any procedure relying on dictionary order can experience all the classes of bugs associated with relying on mutated state. For example, if procedure P1 relies on dictionary order and calls another procedure P2 on a dictionary, might get a new dictionary back in a case where P2 is written in a functional rather than imperative style.

Essentially, relying on ordering requires tracking identities rather than values. Which suggests a second bug. What does it mean now mean for two dictionaries to be equal?

But it is correct in the sense that the insertion order has already changed.

> That's an argument against ever adding ANYTHING to a new language!

Not exactly, because no error will be raised from Python and subtle bugs could be introduced if someone relied on this behavior.

That's not even valid Python.

  SyntaxError: keyword can't be an expression

You want to pass a list of tuples:

  from collections import OrderedDict
  assert list(OrderedDict([('a', 1), ('b', 2)]).keys()) == ['a', 'b']

The problem isn't that .keys() isn't returning them in order. The problem is that the kwargs (a=1, b=2) aren't passed to OrderedDict() in order, so the constructor has no way of knowing which one should be first.

That's exactly the way OrderedDict works too:

https://docs.python.org/2/library/collections.html#collectio...

> An OrderedDict is a dict that remembers the order that keys were first inserted. If a new entry overwrites an existing entry, the original insertion position is left unchanged. Deleting an entry and reinserting it will move it to the end.

Deprecating it won't affect that code since it's more of a flag to indicate that future use should be limited to ensure forward compatibility.

Do you have a link to info about why it's faster? My initial reaction was that the extra indirection would slow it down, but of course that's based on nothing but general knowledge and two seconds' thought.

> Its an idea originally from PyPy afaik.

No, the idea is from Raymond Hettinger on the Python-Dev ML back in 2012: https://mail.python.org/pipermail/python-dev/2012-December/1...

PyPy were the first to bother actually implementing it.

> `OrderedDict` is now just a thin wrapper around `dict`.

It is not. odictobject.c has not been touched at all yet. Also, OrderedDict defines several methods that ordinary dict does not (and unlikely ever will).

Though it's something which has been requested in the past. Also preserving the ordering of class definitions.

Suppose you want to make something like an XML generator helper function:

   def start(tag_, **kwargs):
      write("<" + escape(tag_))
      if kwargs:
          for k, v in kwargs.item():
             write(" " + escape(k) + "=" + quote_escape(v))
      write(">")

(Apply hand-wavying to get the correct code.) This might be called as:

    start("abc", x="1.0", y="2.0")

but generate the output

    <abc y="2.0" x="1.0">

when you want it to be:

    <abc x="1.0" y="2.0">

The output order depends on the Python hash implementation, which (in modern Pythons) is randomly selected during startup. For an API which preserves order, you must currently either pass in the pairs in iterable order, like:

    start("abc", (("x", "1.0"), ("y", "2.0")))

or switch the API to pass in a dictionary-like object instead of kwargs, then switch to an OrderedDict (which must also be initialized with pairs in iterable order).

In 3.6, there's no need for that -- kwargs will preserve the keyword parameter order.

Across processes perhaps? But hash(a) == hash(a) and hash(b) == hash(b) so a dictionary that with a and b inserted will yield the same order every time.

    d1 = dict()
    d1['a'] = 1
    d1['b'] = 1
    print(d1.keys())

    d2 = dict()
    d2['a'] = 1
    d2['b'] = 1
    print(d2.keys())

Deterministic means it is completely determined by the parameters, so it doesn't change from one call to another. This seems to be exactly what you are naming "stable" here. If it outputed the keys in a known order, it would be predictable.

Yet, the GP is just complaining about two different and widespread definitions of "random", so, whatever. It would be great to have a contexts dictionary we could use to resolve such things.

How about unspecified order?

If you care enough to disable one specific CA, then you likely know how caches work and how to find the alternative. Did you plan to post a comment like this on every single HN article which uses StartCom's cert?