Hacker News Reader

Hacker News

Comment by thesmallestcat | Hacker News Reader

Full thread

thesmallestcat

A single class that:

* De/serializes JSON and YAML

* De/mangles, de/encodes keys

* Provides automatic, expensive hashcode

* Blacklists/transforms a bunch of likely keys because they conflict with reserved words

* Overlays attrs (__box_heritage)

* All in pretty complex code that obfuscates what you're really doing (especially to a maintainer)

For the ability to avoid importing json/PyYAML and use clear key lookups? The author must really like JavaScript syntax, or something, because I'm not seeing the point in this layer at all. It's the sort of library that you discover your inherited codebase is using, and just say, "fuckfuckfuck...", because it implies that the author cares more about pushing round pegs into square holes than writing the freaking application code; it reeks of inexperience.

View on HN

I get the itch its scratching. Working with REST services in particular can become very tedious, real fast, without some sort of de/serialization beyond json -> dict.

Many python REST API client libraries end up rolling their own one-off dict -> "dot dict" transformation helpers, for better or worse.

This sort of thing can be great for interactive use-cases, where terseness really helps (notebooks, REPL's etc). I'd definitely use something like this, there. But I would share the same reservations as you in using this library in a serious project.

Why so negative here?

I like Python a lot, and I don't write much Javascript, but one thing I wish I could do in Python is the dot notation from a dictionary. I sometimes used namedtuple as a cheap (but "immutable") class, so I can simply use dot notation when I am passing my object around my functions, instead of always stuffing the data into a dictionary.

"I like Python a lot, and I don't write much Javascript, but one thing I wish I could do in Python is the dot notation from a dictionary"

Why though?

Because it saves me a lot of chars.

foo['name']['attr1']['attr2']['morefuckingattr'] vs foo.name.atr1.attr2.morefuckingattr

More of a personal preference.

Saving chars is often a poor reason to design a language feature.

Nope. It's called developer ergonomics.

The annoyance here is a feature not a bug. Deeply nested dictionaries, in my experience, are a source of infinite KeyErrors. The problem with using a dictionary as a cheap object is that it's very easy for it to be an object you weren't expecting. Don't nest dictionaries in Python!

So then what are you supposed to deserialize nested data structures like JSON into? Dictionaries are the obvious "right" way, Python's annoying tooling aside.

Namedtuples or classes that represent the data, I'd suggest personally.

I would agree. Namedtuple has an asdict method so that reserialization to JSON is simple; you can do the same for a class.

I'd say that dictionaries are only the obvious "right" way if you're not planning on accessing any of their fields. Otherwise you have some set of expected fields, in which case you can define a namedtuple and avoid a vast array of nasty surprises. It's really easy to initialize a namedtuple from a dict (use ), as long as the dict has the right fields. If it doesn't, and you're planning on using its fields, the fact that this raises an exception is a good thing.

What do you think of this library: https://github.com/selik/destructure

It'll enforce schema and destructure the nested dict/list that results from JSON decoding.

That's like asking, "What data structure should I use for this CSV file?"

Depends on the usage. Sometimes you leave them as nested dictionaries and lists. Sometimes you transform them into tabular data. Sometimes you aggregate, etc.

No, it's called being terrible at your job. If you need to add complexity to your project just to save you some typing, you have no concept of what you're doing. It's like an accountant not understanding math.

Complexity can hide in unexpected places and sometimes a dash of complexity in one place can make something in another place much simpler.

I think this library has it's uses. I wouldn't use it uncritically but I wouldn't dismiss it either.

Developer ergonomics does not (always) mean "preventing effort".

If that were true, there would be no type systems.

Part of ergonomics is simplicity about what the thing you're looking at actually is.

True, but it saves a relatively small number of chars (~12) and does so at the same time as:

A) Giving off a signal that you're dealing with an object rather than a dict.

B) Making it cumbersome to swap out some of those selectors with variables.

C) Making it difficult to deal with the attribute not being there (with a dict you can say .get("attr2", {}) and it returns a default.

I understand. I have another motivation which is getting JSON and wanting a OOP object to mainpluate with. Take AWS' boto3 response. Well documented but the nested response structure gives me a chill and I wish there's a direct object out of the JSON. I instead had to write my own class for the conversion. If I could mainpluate dict like mainpluatig attribute in a class then I could just write my function's contract "input is a response object from X APi" instead of "dict of this form."

Quite annoying and I know some people considered what I wish is a bad practice.

That's a tough preference with Python, because of the semantics of what [] and . mean and how ingrained those semantics are in the language. Put another way, the following should hold:

       x.y == x.__getattr__("y")
    x["y"] == x.__getitem__("y")
    assignment == set{attr,item}
    del x.y, del x["y"], etc
    len() and slices of items, not attrs

etc, etc. If you start messing with those semantics, it can become very confusing very quickly in Python and you probably want a data type. Remember that Python gets nervous about cleverness.

One way I've approached your problem before is a recursive helper where I borrowed some concepts from jq:

    val = fetch(result_dict, "foo.bar.baz.plonk[0]")

Because then you can also isolate the missing key handling and all that stuff into your helper, rather than changing the semantics of an untyped dict. Aside from typing the responses from APIs -- the way better option -- I found this a reasonably Pythonic approach toward dealing with the annoyance.

Personally, I only use hard-coded strings like that for shallow nesting. If I need to index 4 layers deep, something down there is going to be turned into an object, otherwise enumerating all the possible keys to ensure the code is correct becomes a nightmare. And once you do that, you can use dot notation anyway.

because of duck typing.

a['b'] = 'c' # means i must use a dict a.b = 'c' # means must be an obj

if they were interchangeable then I could have more reusable code.

You're using named tuple correctly, but why would you want dot notation on a dictionary? There's a reason square bracket notation is used to access dict items. Because it's an item of a dictionary, not an attribute.

Nested namedtuples are way, way better than the proposed approach. I've found that nested dictionaries in Python are fragile in two dimensions: not only do you have no guarantee that the values in your dictionaries are appropriately typed, you have no guarantee that the keys even exist! The whole thing just collapses in a messy heap if you look at it sideways. Furthermore, it's very hard to check to see whether anything has been changed deep within your crazy nested dictionary structure. With nested namedtuples, immutability lets you use "is" on the root to see whether anything has changed deep inside. Sure, you can't say foo.bar.baz.quux = 5, but did you really want to do that, knowing how hard it would be to use that information?

Check out attrs. You can use frozen=True to get the immutable behavior as well, but there are many other benefits over namedtuple.

https://attrs.readthedocs.io/en/stable/index.html

>It's the sort of library that you discover your inherited codebase is using, and just say, "fuckfuckfuck...", because it implies that the author cares more about pushing round pegs into square holes than writing the freaking application code

Ugh, this really hits home.

I just started a new day-job where I'm essentially being asked to un-fuck a python codebase. Your statement neatly captures my feelings when I discovered the presence of hacked-together singleton classes littered throughout the thing.

Good luck unit-testing that...

I think the python implementation of jsonpath is more appropriate for doing this sort of untyped access. If clean dots are really preferred, make classes! :)

I think if 'Box' was broken down into smaller 'ZipLockBags', it would be better; it is doing too much things.

Agreed, I have no idea how this is easier. If anything it makes it impossible to write dynamic code. This is a perfect example of a library that (theoretically) helps the programmer at the expensive of more inefficient code.

> For the ability to avoid importing json/PyYAML and use clear key lookups?

I think the advantage over json/PyYAML is not having to supply conversion functions for Decimal, datetime, or user defined datatypes.

I agree that that's useful, but it should be a separate library, and I'd be surprised if there weren't a few already (I'm guilty of writing one-off serializers for datetimes/dates).

Reply on news.ycombinator.com