Hacker News Reader

Comment by nerdponx | Hacker News Reader

It's a nice idea, but i never like writing what amounts to a DSL in strings in my code (yes, that applies to in-code SQL as well, although that's often unavoidable).

I prefer the `get_in()` method from Toolz: http://toolz.readthedocs.io/en/latest/api.html#toolz.dicttoo...

View on HN

I agree, I don't like the magic string approach (even if it is mostly just dot-notation attribute lookup). However, there is some good stuff here, and nested data lookup when value existence is unknown is a pain point for me.

In addition to the string based lookup, it looks like there is an attempt at a pythonic approach:

  from glom import T
  spec = T['system']['planets'][-1].values()
  glom(target, spec)
  # ['jupiter', 69]

For me though, while I can understand what is going on, it doesn't feel pythonic.

Here's what I would love to see:

  from glom import nested
  nested(target)['system']['planets'][-1].values()

And I would love (perhaps debatably) for that to be effectively equivalent to:

  nested(target).system.planets[-1].values()

Possible?

--- edit: Ignore the above idea. I thought about this a bit more and the issue that your T object solves is that in my version:

  nested(target)['system']

the result is ambiguous. Is this the end of the path query and should return original non-defaulting dict, or the middle of the path query and should return a defaulting dict? Unknown.

The T object is a good solution for this.

I prefer your nested() version. I don't see why your final example would be ambiguous either, if it always returns a defaulting dict which you always have to extract with .values() there is no ambiguity. This is similar to how Javas Optional<T> work. The problem could be that the type checker isn't good enough and writing code like if nested(target)['system'] == "lazer" could pass the type checker.

Objects are overkill. Use an iterable of getitem lookups. They compose easily and don’t have much overhead (cpu or cognitive). From the library I use at work, inspired by clojure.core/get-in, it would look like this:

    def get_in(obj, lookup, default=None):
        """ Walk obj via __getitem__ for each lookup,
        returning the final value of the lookup or     default.
        """
        tmp = obj
        for l in lookup:
            try:
                tmp = tmp[l]
            except (KeyError, IndexError, TypeError):
                return default
        return tmp


    data = {“foo”: {“bar”: [“spam”, “eggs”]}}
    # find eggs
    get_in(data, [“foo”, “bar”, 1])

By using __getitem__ you naturally work with anything in the python ecosystem.

The other problem that T solves and nested doesn't is being able to reuse a spec. Once you have a spec, whether created with T or directly, you can call glom multiple times with the same spec, pass the spec to another function, tc.

That’s called defining a function.

My thoughts also turned to toolz.

Here's a example comparison:

glom

  glom(target, ('system.planets', ['name']))
  # ['earth', 'jupiter']

toolz

  list(pluck('name', get_in(('system', 'planets'), target)))
  # ['earth', 'jupiter']

The real advantage over toolz/get_in is in the breadth of types glom can support (http://glom.readthedocs.io/en/latest/api.html#setup-and-regi...), the extensive fallback behavior (http://glom.readthedocs.io/en/latest/api.html#advanced-speci...), and "T" specifier (https://sedimental.org/glom_restructured_data.html#true-pyth...), which allows performing object-oriented traversals and calls.

Check the post for more examples of where path-based access falls short of the mark!

(Sidenote: I find it very clunky that get_in() takes a "default" kwarg and _also_ a n"o_default" kwarg. Use a Sentinel object! http://boltons.readthedocs.io/en/latest/typeutils.html#bolto... )

Sentinel is one of my favorite objects in Python.

What's the advantage of make_sentinel vs just doing _MISSING = object()?

A Sentinel is guaranteed unique and distinct from all other objects.

I'm fairly sure object() also returns an object which is unique and distinct from all other objects. The only difference as far as I can see is that make_sentinel returns an object that has a unique and distinct type from all other objects, but I don't see why you'd be checking the type of your sentinels in Python.

Hehe, boltons.typeutils.make_sentinel should really be documented better for more experienced developers. A few small advantages: a nice repr, pickleability, and (back to the first advantage really) good rendering in a Sphinx autodoc context. :)

a nice __repr__ is the main thing

Sorry to double reply, but yes the "T" spec is exactly what I would want. IMO it's yet another step better than using a "spec" made of untyped nested lists and dicts.

DSL in strings = bad

DSL in native syntax = good

I feel the same way, although my instinct is generally to build a custom generator. Only costs a couple lines but is plain old python and quite explicit

    target = {'system': {'planets': [{'name': 'earth', 'moons': 1},
                                     {'name': 'jupiter', 'moons': 69}]}}

    glom(target, {'moon_count': ('system.planets', ['moons'], sum)})
    # vs
    def iter_moons(t):
        for planet in target['system']['planets']:
            yield planet['moons']

    sum(iter_moons(target))

would have to combine with `defaultdict`s if your nested data is only sometimes there though

That's a great example! Mind if I use it? ;)

You can also use a list comprehension like so.

>>> sum([x['moons'] for x in target['system']['planets']])

For simple cases like this it doesn't even cost a couple lines as sum() can take a generator expression:

  sum(planet['moons'] for planet in target['system']['planets'])

Combining ideas from parent and grandparent:

  sum(planet.get('moons', 0) for planet in target['system']['planets'])

the only string parsing is 'a.b.c', which is mostly equivalent to T.a.b.c so you can completely ignore that capability

the very slight difference is that using T you must be explicit about attribute access vs key access whereas 'a.b.c' will try both

Why is writing DSLs strictly worse than writing complicated transformations built on top of the limited constructs provided by the language itself? (You said never)

The issue isn't dsls, it's string dsls. Doing it within the language can provide stronger garauntees.

As a simple example, you get some syntax checking.

Precisely. Until my autocomplete, syntax highlighting, and linting engines recognize it, I'll pass.

I think y'all might be getting a bit up in arms about a non-issue. Seasoned glommers (and experienced Pythonists), will almost always be using T: http://glom.readthedocs.io/en/latest/api.html#object-oriente...

While mentioned in the post, it's not front-and-center because that's the sort of Python superpower that can look scary to less-experienced devs.

They’re pretty comparable when the strings are constants. The fun begins when the string is dynamically created.

I wouldn't recommend dynamically generating strings. Strings are just a shorthand for Path objects, use Path instead: http://glom.readthedocs.io/en/latest/api.html#specifier-type...

I definitely wouldn't recommend it either. Sorry not to be clear.

In my experience, it's almost impossible to dissuade people from generating strings if the API affords it. We can't even stop people from generating SQL by concatenation!

Musing: does Python have a way (Mypy?) to declare that a method can accept only constant expressions?

Reply on news.ycombinator.com