Precisely: Better assertions for Python tests | Hacker News Reader

Precisely: Better assertions for Python tests

github.com

60 pointstrymas29 comments

It would be great if they could go into more detail on what they've improved over pyhamcrest (https://github.com/hamcrest/PyHamcrest) which I do use on occasion when there is no other easy way to write a good matcher. That said though, pytest's assertion rewriting goes a long way towards making hamcrest-style assertions redundant. But not completely.

Do you have an example of the "not completely" case?

It's still not that easy to assert on some of the keys in a dict but not others, especially if you only want to assert on some of the keys nested inside that key, and so on. There are a few other cases that perhaps don't come to mind straight away, many to do with having complex rules for what is allowed to be in another thing.

I'm not sure what do you mean exactly... If it's something like:

    assert foo["bar"] == 123
    assert foo["baz"]["bam"]["boo"] = "abc"

I don't see how is that made easier (and less ambiguous) in hamcrest-style assertions...

The case is more like when you want to assert against some of the contents of a dictionary but not all (say 60% of a 30 key dict), and not all those assertions are based on == (and it's a multilevel nested dict).

There's no particurly terse why to write an example I'm afraid.

(I'm the author of precisely)

Better error messages was the motivation: PyHamcrest tends to put everything onto one line, which makes it hard to discern structure. In contrast, precisely tries to produce an error message that uses line breaks and indentation to match the structure of the matcher, hopefully making it clear exactly which part of the matcher has failed.

That sounds like a feature PyHamcrest could add. Did you try adding it there?

pytest assertion rewrite in the standard Python library would be really, really nice. I don't think the standard lib gets enough love.

Why choose preceisely over

    assert all([el in result for el in ['A', 'B']])

There is definitely room for more brainfarts in the above, but introducing a new library with its own semantics into a team also have its costs.

This example doesn't test for absence of any other elements, so it's not as precise.

That said, I largely agree with you - checking for this exact example from the README file works much better (and IMO more readable) as:

    assert set(result) == {"a", "b"}

The precisely example doesn't even check if the result is a list, so it can be anything.

That is better, but doesn't check for incorrectly repeated elements:

    result = ["a", "b", "b"]
    assert set(result) == {"a", "b"}

In fact that is the first main motivating failure case discussed in the readme. But actually that's simple enough to deal with too:

    from collections import Counter
    assert Counter(result) = {"a": 1, "b": 1}

But would be trickier for mutable / unhashable types (not sure if "precisely" can deal with those).

Yup, precisely should work fine with mutable / unhashable types -- it just relies on equality to check the elements in an iterable.

More specifically, writing:

    contains_exactly("a", "b")

is equivalent to:

    contains_exactly(equal_to("a"), equal_to("b"))

To avoid repetitions, can you do this?

    assert sorted(result) == sorted(['a','b'])

I think that only works if `result` is a list

Nah, sorted can be used on any iterable

Ah that's better than my collections.Counter trick. Yours only needs the items to comparible rather than hashable, which is usually a looser requirement. (In principle you could have an object that's hashable but not comparible I can't immediately think of any).

The main problem is probably heterogenous collections. If you have a collection with both ints and strings then hashing will work fine but comparing will fail (in Python 3).

(Frozensets are hashable but not sortable but that's probably more niche.)

Or even better (since we're testing against an oracle value anyway):

    assert sorted(result) == ['a', 'b']

True you can just sort your oracle values once/in your head this is a valid point. Reasonable to prefer this way!

Thanks for pointing out my brain fart :)

If the author of the library is reading, I think it would be a great addition to have a section discussing the benefit vs writing your own one line checkers.

One of the biggest benefits of libraries like this is that they tend to explicitly state the test author's intent.

To take your example from up top: If I'm coming to the code later, and I notice that it's not asserting for the absence of anything, it may be very difficult for me to tell whether that behavior is your intent, or just a brain fart.

It's just another layer of confusion in the many layers of confusion that lead to test rot.

The error messages in the readme look a lot better than what you'd get from doing a ad hoc solution. That's worth a fair amount.

You run the asset statement through pytest, which patches the generated bytecode to print a good error message

https://docs.pytest.org/en/stable/

(I'm the author of precisely)

If you were to ask me "Should I use precisely in my tests?", my answer would be: it depends. The main benefit is to better describe the intent of your test, so that the assertion is neither under- nor over-specified, with the intent directly stated.

Assuming your assertion is meant to be an alternative to the example in the README:

    assert_that(result, contains_exactly("a", "b"))

I'd suggest that the above states the intention of the assertion, rather than how you check it. For instance, your assertion would allow duplicate elements, whereas the assertion as originally written would suggest that this isn't desired. As other comments have pointed out, you can do things with sorted, Counter or set (depending on exactly what you want to assert), but why worry about what trick to use when you could just directly state your intention?

The assertion using precisely is also (arguably) easier for a reader to know what is (and isn't) being asserted in the test, and makes the test less brittle since you're not accidentally asserting more than intended (for instance, it's common to assert equality with a list, even though you don't actually care about order).

Another common case is when you want to make assertions on a collection, but equality would check too much. For instance, suppose you have a function that fetches users from a database. The fetch can return the users in any order, and you just want to check the names of the returning users, so you can write something like:

    assert_that(result, contains_exactly(
        has_attr(name="Alice"),
        has_attr(name="Bob"),
    ))

How would you write something that means the same thing without precisely? The order isn't deterministic, so you can't write something like:

    assert result[0].name == "Alice"
    assert result[1].name == "Bob
    assert len(result) == 2

We could sort the users by name before making the assertion:

    result = sorted(result, key=lambda user: user.name)
    assert result[0].name == "Alice"
    assert result[1].name == "Bob
    assert len(result) == 2

Personally, I prefer the precisely assertion!

What about something like an equality assertion?

    assert set(result) == {
        User(id=1, name="Alice", email_address="alice@example.com"),
        User(id=2, name="Bob, email_address="bob@example.com"),
    }

Now we've over-specified our test -- we need to know irrelevant details like the ID and e-mail address of the users, which might change and break this test even when the functionality we care about still works. We'll also break the test if we add any more attributes to users.

As you've mentioned, there's a cost to learning precisely. Even if you're familiar with the library, then you're still potentially writing a more complex assertion (where more can go wrong) than just (for instance) an equality assertion. In my experience, on many projects, the ability to state the intent of assertions precisely has far outweighed the downsides, but that's from a position of already being comfortable with the library.

The original title is "Precisely: better assertions for Python tests", and it's far superior to the one submitted here.

Unfortunately that is a misleading headline since these are not better assertions for Python tests. These appear to be an attempt to infect Python code with pointless verbosity found in AssertJ or Hamcrest for Java.

unittest already has a lot of really helpful assertions - I just wish they were more accessible outside of a standard class-based tests. https://docs.python.org/3/library/unittest.html

I tend to try and lean on the language as much as possible before pulling in libraries like this. This is the sort of thing that you end up regretting a year or two later when the author abandons the project and you can no longer get help for bugs. Unlike vanilla language assertions, you won't need to rewrite your entire test suite when that situation inevitably comes up.

pytest will give you really awesome feedback with vanilla/basic 'assert x in y' statements

Many years ago I wrote https://github.com/elifiner/affirm which replaces the built-in assert statement and provides similar (?) benefits without changing the syntax.

Reply on news.ycombinator.com