Opnieuw: A simple and intuitive retrying library for Python | Hacker News Reader

Opnieuw: A simple and intuitive retrying library for Python

tech.channable.com

71 pointsarianvanp45 comments

What's the difference between this one and tenacity?

At least a good readme, which is always a good sign.

From the article, they claim that the parameter names are easier to understand and therefore easier to enter correctly. As I've not used either lib directly, I'll have to take their word for it.

Personally I would avoid this kind of npm-esque micro-dependency in my Python projects. I’ve written essentially the same decorator for many of my projects; it takes ten minutes and a few dozen lines of code.

Would you really want to roll your own exponential backoff w/jitter for every project though? It's not the easiest to get right, and since you're looping in a somewhat complicated way, it's definitely prone to accidental infinite loops in my experience.

Retrying seems perfect for a library to me. For instance, Haskell has a popular retry library https://hackage.haskell.org/package/retry-0.8.1.0/docs/Contr... and I wouldn't consider it bad to just bring it in.

Is this more symptom of Python deps being "riskier" than Haskell deps due to types or versioning or something?

Any dependency is also a risk. What if the next version does not just contain a nice feature that you want but also a nasty and subtle bug that you notice only later? Also, maintenance of a small library is by no means guaranteed to remain at the same level of quality. Generally, one is much safer with larger and more serious libraries. I would say one should just have dependencies that provide some rather serious set of features that would cost a lot of time to implement oneself.

Not at all, I think they just have some notion that npm = bad and npm = (small) deps, so (small) deps = bad.

I agree, it's harder to get right that it seems from first glance. The number of implementations I've seen with broken or not-honoured timeouts...

Python also had a retrying library; apparently it's no longer maintained. I've used the tenacity "fork", which has also evolved/introduced some great features like combinable and chainable wait/backoff.

> Not at all, I think they just have some notion that npm = bad and npm = (small) deps, so (small) deps = bad.

No, I don’t have that notion. Apparently not every package on npm is small or trivial, but the abuse of trivial packages is most well-known and disastrous in the npm ecosystem.

Edit: My god, iOS autowrong is all the rage today.

This library posted here is only ~150 loc without docstrings, and it’s more complicated than my typical equivalent decorator factory with basically the same features, sans logging.

I just don’t find this tricky. If only every problem I need to solve is as simple as this one.

> Is this more symptom of Python deps being "riskier" than Haskell deps due to types or versioning or something?

No, it’s a distaste for tons of small third-party packages that could easily be done in place. If anything I found Haskell deps more prone to breakage with upgrades compared to Python deps (given my limited experience with Haskell), at least with the set of popular packages I tend to use. Part of that is Python packages tend to not depend on a ton of other packages.

By "breakage with upgrades" do you mean bugs introduced due to compiler/library bumps or just PVP constraint violations?

The latter — not necessarily PVP constraint violations though, but rather dependency hell situations. One example that pops to mind: packaging git-annex for Homebrew was always “great fun” until we settled on resolving with LTS Haskell (IIRC).

The thing that always gets me with retries is error handling.

What errors should be considered retryable, vs raise immediately?

Do you log a retryable error? I've had systems mysteriously "hung" before which on closer inspection were infinitely retrying with a retry library that didn't log anything...

And if you do exceed your max retries, which error do you raise? The first one? The last one? There's no guarentee they'll be the same. A completely different "TooManyRetries" error?

These are the concerns I'd like to see addressed in a high level overview of a library like this one.

Also: what if you get different retriable errors every time you re-try: should you have different retry count for each distinct retriable error? When you finally give up, should you report _all_ of the distinct errors or just the earliest or latest retriable error, or ...

Would be interesting to read more specific comparisons with existing mature libraries, e.g. backoff [0]. Feeling I get from reading the motivation behind a new library is that most of it can be achieved by using backoff and perhaps a custom delegating decorator that remaps parameter names and does extra checks if necessary.

[0] https://github.com/litl/backoff

The title is a dutch word, in English translated to "again" or "repeat".

Yes, but it's one of those words, where pitch and intonation (and -in this case- past memories of dread) are as important as the actual syllables.

The vowel combinations always trip up foreigners. Nieuw is Dutch for new and it is pronounced essentially the same. The ew sound basically transliterates to ieuw. Op (on) combined with Nieuw, basically means "on new", or 'again' in proper English.

I'll be the pedant and argue that if you want to figure out how to sound proper Dutch, there's a difference.

"new" sounds like "neeuuuuu" in US English, and a diphtong "nuuu" in British English.

"nieuw" sounds like "niii" followed by a very short "oooo" as in "do" followed by a slightly more pronounced "w" at the end.

I'm aware that there are proper ways to codify pronunciation, but it's more fun this way and I know nothing of it.

Still seems a decent company. Nice that you take the time to properly deal with technical problems instead of just doing the whole 'plough forward even though you're just ploughing air' thing that happens just about everywhere.

I don’t have a better solution but I was taken back by the length of the very verbose / descriptive param

retry_window_after_first_call_in_seconds

Am I the only one that thinks that is a bit excessive?

If you're using something like this, then the details of when, what and how you retry is probably something you care about, so having the library be explicit about it and making it possible to know what something means without having to look at the docs seems sensible to me.

This is some Java-level explicitness. Maybe, drop the `_seconds` suffix and replace it with something like `retry_window_after_first_call = '60s'`?

I agree it might look a bit excessive, but it greatly cuts down on complexity and possible unexpected behavior.

If you accept '60s' as 60 seconds, then the next questions are:

- Will it also accept '60000ms'?

- What does it do if I pass an int or float anyway? Will it implicitly take it to be seconds?

- Will it throw an exception if I use an unexpected value? If so, which exceptions can I expect? Will it just chug along and use a default?

Something I picked up from another developer is to use timedelta for these type of parameters. It avoids hoping that you get the right granularity for everyone.

   retry_window_after_first_call=timedelta(minutes=3.5)
   retry_window_after_first_call=timedelta(days=2)
   retry_window_after_first_call=timedelta(minutes=50)

Nice! Yes this is the better option. IMO, should just be

  retry_window=timedelta(...)

60s. was wondering how we could add physical units to python? highjack the number literals and variables like this: 1.s, pi.radians? or maybe the way golang does it: 1 * time.seconds?

would be nice to have

Sounds like a fun hack, I encourage you to give it a try and then never EVER use it in real code :P

There's this: https://pint.readthedocs.io/en/0.10.1/

I also found this package: https://github.com/sbyrnes321/numericalunits

Which uses the interesting approach of setting all "units" to random floats: A complete set of independent base units (meters, kilograms, seconds, coulombs, kelvins) are defined as randomly-chosen positive floating-point numbers. All other units and constants are defined in terms of those. In a dimensionally-correct calculation, the units all cancel out, so the final answer is deterministic, not random. In a dimensionally-incorrect calculations, there will be random factors causing a randomly-varying final answer.

Which presumably is done to avoid overhead from carrying around units with each number, but forces you to run calculations multiple times (with different random seeds) to verify the result is correct.

While the idea of '60s' is interesting, in lieu of that I always err on the side of explicitly specifying the time units when the argument provided is just a number. Not doing so has bitten me too many times.

yes it is excessive!

If you are not familiar with how retrying is used and why exponential back off, jitter, etc are useful here is a shameless plug to a blog post I wrote recently https://blog.najaryan.net/posts/autonomous-robustness/ (the first half pretty much is about all these things).

I like the name. Maybe this should be merged into requests at some point?

We did an internal naming competition for the library and one of the suggestions was "retreitz - Retries for humans".

En jullie hebben een kantoor in Utrecht! Gezellig.

(And you have an office in Utrecht! Cozy.)

So that's why you guys speak Dutch anyway!

Do you speak Dutch? It's like the Python Zeep (SOAP - API protocol) client is Dutch for soap, Opnieuw is Dutch for again.

I always wandered if that was a nod to Guido Van Rossum that people like naming python packages in Dutch?

Channable is a Dutch company, and a lot of people who work here speak Dutch. I don't remember where we got it from but it is possibly a nod to Zeep.

I joked that we should name our library such that you would have to do

  from driemaal import scheepsrecht

"There should be one-- and preferably only one --obvious way to do it.

Although that way may not be obvious at first unless you're Dutch."

-- Tim Peters, The Zen of Python

O:-)

I believe at least one of the contributors to Opnieuw is Dutch.

No, but I speak Afrikaans, which is closely related to Dutch. So "opnieuw" is easily understood. In Afrikaans it would be "weer" (or "weer oor").

Most English names are already taken on PyPI. Using another language is one way to get around it.

I think I’ll try this, but for devs:

a simple while True + try/catch + time.sleep(n*2) will do exponential backoff pretty easily.

But no jitter, and no timeout.

The post clearly states the problems they were trying to solve, none of which are taken into account here (consistency, ease of parameterization,..) It seems like people are reading too many "X in Y lines of code" posts :D

https://github.com/invl/retry works well for me. only problem is that logging issue that's still open i believe.

seems to me that the authors wrote their own library instead of trying to make the ones that didn't meet their need better. well, they never mentioned they tried...

anyways, some of their choices like that long variable name are odd to me. i can understand the unit problem though. golang solves that so nicely!

Reply on news.ycombinator.com