Hacker News Reader

Comment by NoboruWataya | Hacker News Reader

I suspect people will use these comments to share their wishlist of Python features, so let me add that I really wish Python had a safe navigation operator, for when you are dealing with nested objects that could be None. I have been trying to parse a lot of XML and JSON in Python lately and a feature like that could really help reduce boilerplate checks.

https://en.wikipedia.org/wiki/Safe_navigation_operator

View on HN

You can participate in the latest iteration of discussing this feature here: https://discuss.python.org/t/introducing-a-safe-navigation-o...

The syntax has a bit of a learning curve, but there's a library called glom that can be helpful in parsing heterogeneous JSON. Specifically it has a coalesce operator that let's you define multiple paths to get the data you are trying to get at, with a default value if none of the paths are valid.

https://glom.readthedocs.io/en/latest/api.html#defaults-with...

e.g.

    target = [{"a": {"b": "c"}}, {"a": {"c": "e"}}, {"a": {}}]
    results = glom(target, [Coalesce("a.b", "a.c", default="")]) # -> ["c", "e", ""]

I believe PEP505 should cover what you're asking, but from what I can tell it seems stalled out.

https://peps.python.org/pep-0505/

I haven't even used Python for like four years now, but all of these suggestions to do try except and get.key() or {} seem to be ignoring that Python has had a defaultdict in the standard library for, I don't even know, at least the last decade and a half, as long as I've been paying attention. It's even written in C: https://github.com/python/cpython/blob/d9246c7b734b8958da034....

defaultdict doesn't seem to easily solve for the main topic of deeply nested values.

For JSON there is the `jmespath` library which might help.

https://github.com/jmespath/jmespath.py

    print(jmespath.search(
        expression="some.deep.nested.value",
        data={"some": {"deep": {"nested": {"value": 2}}}},
    ))
    prints 2

    print(jmespath.search(
        expression="some.deep.nested.value2",
        data={"some": {"deep": {"nested": {"value": 2}}}},
    ))
    prints None

While not as short as a proper operator .get("key", {}) does work.

Of course this only works at one level and not arbitrarily deeply, but you still need to check for None with this code; it may actually be better to write `x.get("key") or {}` so that you always get an empty dict.

I write 'may' because the difference between None and an empty dict may be very subtle and rarely does the API specify with enough precision what are supposed to be the semantics of each case.

> Of course this only works at one level and not arbitrarily deeply

It does though, you can chain as many of these as you want:

    some_dict.get("level_1_key", {}).get("level_2_key", {}).get("level_3_key", {})...

edit:

>> but you still need to check for None with this code

> Not sure what you mean here. You only have to check for None if you use `.get("key")` and don't provide a fallback value.

GP was talking about `{"foo": None}` and trying to drill deeper, which I misunderstood. Still, a simple try/except allows you to short-circuit the deeply nested access.

The issue is if the dictionary has a key defined and the value is None. Your get expression will return None, causing your next access to raise an error.

Ah, I see. Still, I'd just catch that exception and move on because it's obvious you won't have anything more deeply nested anyway. Blowing up on `None` is an easy short-circuit.

    >>> foo = {"foo": None}
    >>> print(foo.get('foo', {}).get('bar'))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    AttributeError: 'NoneType' object has no attribute 'get'
    >>> print((foo.get('foo') or {}).get('bar'))
    None

Right, see my other comment:

> Blowing up on `None` is an easy short-circuit.

The thing being discussed is attempting to access deeply nested values, so short-circuiting here is a win-win. I.e., you wouldn't want to unnecessarily traverse tons of empty dictionaries using the `.get() or {}` trick.

> x.get("key") or {}

This has the side effect of replacing falsy values like False or 0 with an empty dictionary instead of giving you the actual value. For the exact intended behaviour, you do need to explicitly check for None instead of using a short circuit trick with or.

Yeah, when trying to drill down into deeply nested structures, it'd be a waste to just blindly nullish-coalesce any None objects into empty dicts instead of just immediately short-circuiting out of the deep access. It's a cute trick but would not pass code review in my shop.

The problem was underspecified, but I suggested that under the assumption that the value would be an Optional[dict]. If you have another falsy value, it means either Optional[Any] or something like Optional[Union[dict, list]] (I put list as an example, but you get the idea).

I would say the only reasonable choice is Optional[dict], in which case it should be sufficient, but Python being Python, it could be anything. And in these cases you need to handle all cases way more carefully.

    def safe_navigation(obj, path: List[str], default=None):
        try:
            for key in path:
                obj = obj[key]
            return obj
        except KeyError:
            return default

Yeah. I thought it would just be a try-except, but that turned out pretty ugly.

But really, how do you know if your path is missing or the object at the end of the path is None, without using exceptions for communicating this?

I would write:

    def safe_navigation(obj: Any, *path: str, default: Any = None, strict: bool = False) -> Any:
        sentinel = object() if strict else None
        for key in path:
            obj = obj.get(key, sentinel)
            if obj is sentinel:
                return default
        return obj

The only new disadvantage of this is that it only works on mappings and no longer works on sequences.

But I really don't like having this much `Any`; it's generally a sign of poor design.

I wouldn't call a function that mutates the input "safe navigation", personally :)

Er, it doesn't? `.get()` (or `[]` in the original) doesn't change the object, and `obj =` only changes which object the local variable refers to?

This isn't like C++ with its intrusive `=`.

You're right. Python is pass by reference and it's the reference being changed, not the object. This is what happens when you read code on the internet after drinks.

There's certainly humor in a function called "safe navigation" mutating the data structure I'm supposedly safely navigating through :D

I don't think this is correct . The name obj gets rebound, but the dictionary being traversed isn't mutated in any way.

Yeah I agree, no mutation here as far as I can tell

Correct. I was drunk.

I did type it on my phone, so I’m sure it needs a little correction (particularly the choice of exceptions is not optimal, I think) but feel free to run it in any recent python interpreter and return back with any mutation you see.

    $ python3
    Python 3.11.6 (main, Oct  2 2023, 20:46:14) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from typing import List
    >>> def safe_navigation(obj, path: List[str], default=None):
    ...     try:
    ...         for key in path:
    ...             obj = obj[key]
    ...         return obj
    ...     except KeyError:
    ...         return default 
    ... 
    >>> obj = {1: {1: {1: 2}}}
    >>> safe_navigation(obj, [1,1,1])
    2
    >>> obj
    {1: {1: {1: 2}}}

It's been stuck in discussions for a long time, unfortunately. The main reason that I remember gleaning from discussions is it's unclear just how "special" None should be treated as, and whether having special syntax for it is excessively "special".

I saw some suggestions for overridable behaviour in the data model for None-like objects, but didn't scrutinise them closely. Either way, it seems controversial. On the one hand, the ability to override operators is really key to Python's design, but on the other it could lead to rather surprising behaviour from overeager overriding. (But to be fair, this is true of other, more common operators too.)

So, for now we're stuck with `if _ is not None else`, I suppose!

I created a "NoDict" class that does this awhile back: https://github.com/BlackEarth/bl/blob/main/bl/no.py

That's something I've wished for a long time that it had. I know you can use exceptions to get similar behavior but it's really ugly and I just generally don't like raising exceptions for cases that aren't actually exceptional (even though I know Python uses exceptions for normal flow with StopIteration).

What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator. To me the safe navigation operator goes hand-in-hand with that.

> What's actually a bit odd about Python missing that operator is that its `or` operator acts as a rough equivalent of C#'s null-coalescing (`??`) operator.

Python `or` is C# `||` plus type coercion. It is not even roughly `??` because important non-None Python values are falsey (0, False, empty collections).

    > python3 -c 'print(None or "hello")'
    hello

It evaluates to the left if that's truthy otherwise it evaluates to the right, similar to how `??` evaluates to the left if that's non-null otherwise the right.

In C# even if `||` automatically coerced the types, the output will always be a bool.

On a library level you could return an "empty" object which you could call methods on, returning empty objects and so on, for chaining.

It seems that every language, given enough time, wants to tend towards Haskell!

Python, prouding (← is that a word?笑) itself to be the foremost language of duck typing, I'd really want to see that too.

'priding' is idiomatic English

Priding ;)

touting

I use Pydantic for this for JSON.

Reply on news.ycombinator.com