Solving Algorithmic Problems in Python with Pytest (2019)

Depends on the ISP, my mobile dongle has a dynamic public ip, I have a little twitter bot that sends out a tweet every time its ip changes.

On my wired connection I have 1tb symmetric fiber with 16 static ipv4 ips and an arbitrarily large number of ipv6 addresses, I just need to register more if I need them (which I really don't).

That said I didn't get the job, so caveat emptor.

> Putting randrange as an argument creates deeply worse complexity issues and externalizes a needless burden on your user who must carry around their rand function and must understand what type of return value format the passed-in function has to give in order to be usable in the internals of findBias. It’s shocking to me you think this is somehow simpler or that the testability is somehow better.

It gives users the option to do that, but doesn't burden them. This can be an implementation detail, whilst the public API provides a default like `bias = lambda: findBias(random.randrange)`. Since it's Python we could also use a default argument, e.g.

    import random

    def findBias(randrange=random.randrange):
      heads = tails = 0
      for i in range(n):
        if randrange(100) < 100*p:
          heads += 1
        else:
          tails += 1
      return heads/tails

> This is how nasty convoluted API get created. Now other functions up the call stack will also have to have a randrange param if they have to pass it forward.

This is overly-simplistic and disingenuous:

If other functions don't care about this parameter, then they don't need to use it, since it's just an implementation detail; they'll call the public API, which will use the default random.randrange, and these callers don't even have to know that the parameter exists. This scenario is strictly an improvement to what you describe; heck, we could even mock 'randrange' like you describe and it would work in exactly the same way! (except we don't need to, since we can just do normal dependency injection via function arguments instead)

If other functions do care about this parameter, e.g. because they want to pass along a parameter from their own callers, then that's solving a different problem. In this case it's even better to be passing around parameters: mocking is a last-resort crutch when we're unable to refactor legacy code, but even in those cases it should only be used during testing. It's a very bad idea to be monkey-patching the standard library during the normal course of a program run; especially when it's just to avoid adding an extra parameter to a function we control.

> Your change is strictly more confusing, has strictly worse coupling and has a strictly worse API and even after all that, the test is not simpler and isn’t even less code than a one-liner mocker.patch in pytest.

I know a lot of this is subjective, but I'd like to point out that the mocking solution essentially works using mutable global variables. If we override the name "random.randrange", we don't actually know what else we might be affecting; for all we know, our bias-calculator could invoke a bunch of helper libraries which happen to rely on certain behaviour from random.randrange, which mocking will break in unexpected ways. In contrast, changing which value we pass as a function argument will not break arbitrary other code far-away which just-so-happens to be using the previous value. That's the fundamental problem with mocking, and why it should be avoided except as a last resort: it forces us to make big, inappropriate assumptions about not only the code we're testing, but also everything else it might ever interact with. The way pytest implements that mocking (by mutating globals like 'random.randint') just makes this even worse.

assert test_performance > threshold ? This would only be a sanity check for big deviations though. For more subtle changes one may need to do a proper statistical test. I believe the jury is still out of how to do that properly for deep ML models - challenge is the lack of independence in CV folds and generally the compute time it takes to evaluate.

However, I would probably not do performance check inside a unit-testing framework. Instead treat this as quality indicators like performance benchmarks, code coverage etc. It may be a "gate", that needs to pass to allow a new model into production.

To evaluate performance over time, one would preferably want labeled datasets for test gathered at different points in time. Which requires a (reliable) continuous labeling process. One can also gather customer feedback about performance, track those as metrics. These things are probably more in the "monitoring" part of a system, rather than unit-testing time though.

You can test on a tester of use some of the examples from the checklist paper. In that paper they might add "I hate you" to some random data and assert that sentiment doesn't improve.

Those are statistics/ML questions, not software development ones, so software development processes like TDD are only tangentially relevant really.

But in any case, it's actually fairly easy to test that your trained model has sufficient accuracy: choose a metric, choose a threshold for said metric, and check that the observed metric on a testing set (data that the model was not trained on) is above the desired threshold. Repeat for several different metrics for a better understanding of how well the model performs. This can be put in a set of unit tests.

Check residuals, inspect the logical implications of the regression coefficients (or whatever), plot a few curves etc to be more sure again. This can't really be put in unit tests, nor should it be. But again, this is more statistics than software development.

Same goes for model deterioration - every so often you check that the metric(s) still beat the minimum threshold on more recent data.