More Gotchas of Defer in Go, Part II | Hacker News Reader

More Gotchas of Defer in Go, Part II

blog.learngoprogramming.com

105 pointsinancgumus53 comments

Also read the official blog post on defer: https://blog.golang.org/defer-panic-and-recover

It covers about half of these gotchas by laying out how the defer statement works in plain English. The other half (like how closures work inside loops in Go) are covered elsewhere in the language tour.

I do really like the visualizations! Makes it very clear how these mechanics work.

The content is good, and it is accurate, but even though defer can be a bit tricky this blog post series seems to literally be describing the entire semantics of "defer" as "gotchas". It's not that tricky!

Well, it depends. Upgoing posts will go deeper on defer.

Thanks! That post definitely a must read.

Btw, the first part was more advanced (but again for beginners and intermediate gophers) -> https://blog.learngoprogramming.com/gotchas-of-defer-in-go-1...

Every point mentioned was not a gotcha to anyone who read the official introductory tour of the language. The behaviour may be unexpected to newbies, but if you spent an hour or so learning the language, you shouldn't be falling for these.

Check out the first part too: https://blog.learngoprogramming.com/gotchas-of-defer-in-go-1...

I fell to these when I was learning Go for the first time and then I've seen many people also did so. There is no rule like these are gotchas for everyone.

Completely agree. This is probably the most comprehensible list of programming language « gotchas » i’ve ever read. It feels really a confirmation that the language authors have succeeded in their goal for simplicity.

I think most of the people who understand the basic principles of how computers (and many languages) work wouldn’t be surprised by any of the „gotcha” described in this article. It seems to be written for people who just started working with golang without any or little knowledge about information technology.

The semantics of many languages' for loops are explicitly designed to avoid this problem: for example, for-of in ES6. Generally, modern language design considers the interaction between loop iteration variables and closure captures. It's reasonable to criticize Go as a language for not addressing this problem rather than users.

I am not sure about this. Swift for example also has `defer` statements but its behavior differs from Go's. Swift executes defer statements at the end of the block, not the end of the function. And AFAICR Swift doesn't evaluate the parameters right away like Go does. If you have already read how defer works in detail in Go, you probably already know this. Devs new to the language or those that haven't used defer in those cases might still be surprised when this shows up.

In only surprised because so many of them are such well-known things to avoid when designing a language and runtime.

Defers just feel like a watered down version of what you get with good scoping and RAII. They're a half measure for something programming languages solved decades ago.

Go has a garbage collector. RAII is only viable in languages where destruction is predictable, like C++ or PHP.

"defer" is effectively used for predictable destruction in Go, and a more automated RAII scheme could be used in its place.

C++/CLI is a language that targets a garbage-collected runtime, yet has full-fledged RAII semantics (they're mapped to CLR Dispose pattern).

Oh, I totally forgot about C++/CLI! You're absolutely right; that's my mistake. I do agree that RAII is preferable.

The tracing GC certainly complicates things, but you can make it work with a "scope" keyword like in D.

PHP also has a garbage collector.

Swift and D also have garbage collectors and support RAII.

And since I already see it coming, reference counting is a garbage collection implementation algorithm in computer science literature.

Yep, and I even referred to reference counting earlier in the week as garbage collection, so I'm going to assume I was having memory corruption with my answer. :-)

Hehe :)

#4 can also be fixed with:

    for i := 0; i < 10; i++ {
      i := i  // Creates a *new* `i`
      defer func() { … something with i … }
      go func() { … something with i … }
    }

This is too sneaky and seems like poor form.

I'm curious what good use-case there is for this, that the language bothered to support and allow this to even compile.

Duplicate variable name declaration + referencing in the same scope is unintuitive at best, and just seems wrong.

Shadowing can be useful if you want to make sure the "old" variable is never referenced again in the same scope.

The reason why this idiom feels "weird" is that the loop construct of the language really ought to automatically make a fresh instance of the loop counter for each trip through the loop.

I don't feel that it's in more poor form than the solution suggested in the article:

    for i := 0; i < 3; i++ {
      defer func(i int) {
       fmt.Println(i)
      }(i)
    }

This shadows `i` pretty much the same amount as what I wrote. If `i` gets a different name inside the lambda, then it'd also be worse because then you could accidentally use `i` still.

I personally prefer the version presented in the article, but your version has at least one nice feature in that you can cleanly specify at the top of the loop body which variables are being shadowed.

It would be nice if Go made a new i for each iteration of the loop.

Lua 5.0 made just one loop variable per loop, like Go apparently does. When I first ran into this behavior, I thought that, although it wasn't what I expected, it made at least as much sense as what I was expecting. Since then, every single time I've been in a situation where the two ways of doing it gave different results, I've wanted the "separate variable for each iteration" behavior, so I was glad when Lua's creators changed loop variables to work like this in Lua 5.1.

(Before this change I would just do the Lua equivalent of your "Creates a new `i`" line.)

C# also switched to a new i for each iteration.

Crikey! You wouldn't wanna allocate closures in a loop anyway, would you? Even if the compiler is smart about it. Doesn't read so well either.. what's wrong with adding a simple argument to your anonymous-func allocated outside of a loop? It's readable, and there are no 'gotchas'. The args are evaluated at the point of defer/go, not func execution. Simples.

Shadowing bites you sooner or later if it becomes a habit and gains you almost nothing in real terms. I keep running into subtle nasties in coworkers' commits from quick'n'dirty-that-stayed "convenience" shadowings. Need I say `err`.. of course linters help, but aren't a given in a "bring-your-own chosen dev-env" team. =)

> Crikey! You wouldn't wanna allocate closures in a loop anyway, would you?

Very often yes I do.

Go encourages synchronous APIs, making it up to the caller to add concurrency. This is great, in my opinion. E.g. this is a common pattern:

  var wg sync.WaitGroup
  ch := make(chan int, len(items))
  for _, item := range items {
    item := item
    wg.Add(1)
    go func() {
      defer wg.Done()
      ch <- doSomething(item)
    }()
  }
  wg.Wait()
  close(ch)
  for res := range ch {
    …
  }

Similar patterns with defer, although yes I'd say less often in a loop. Though in order to be "exception safe" (panic safe) I often do:

  foo, err := openFileOrSomething()
  defer foo.Close()
  [… do something …]
  err := foo.Close()

So that even if "do something" throws exception… err… panics… the file gets closed. And double-closing is safe. That's not a closure though, in this example. So maybe not so good.

None of your explanations/examples counter the fact that you could also declare your anonymous-func just above your loop in a local, ie. `doit:= func(item...` and then simply `go doit(item)` in the loop. You get: a leaner terser loop to later have to read through, no iteration-scope "gotcha", no need to elaborately spell out explicit shadowings for what are naturally semantically really func "args" already/anyway, at worst identical cost (or better)..

But well, guess it comes down to subjective stylistic preferences here =)

> So that even if "do something" throws exception… err… panics… the file gets closed.

Your defer as placed in your above example is already scheduled to run always, even on a later panic. (After all, how else could one `recover` from a `panic` if it wasn't for `defer`?) I don't see the point of the double-closing at all here..

Yeah, I could declare it above. But it's subjectively harder to read, having to jump around. Like you say: subjective.

> Your defer as placed in your above example is already scheduled to run always

Yes, brainfart. Sorry. This is from a completely separate recommendation to defer a close (dropping error return), but also manually close so that closing errors can be surfaced. Matters for e.g. writing files (not so much reading), especially when you don't flush manually.

I code in go every day professionally. These are not gotchas, they are well defined behaviors.

Check out the first part too. Btw, gotcha means what you say: "a gotcha ... works as documented but is counter-intuitive and almost invites mistakes ..." https://en.wikipedia.org/wiki/Gotcha_(programming)

It's indefensible that defer works on the function and not on the scope.

Well I think the function scope the most useful, but wish there was block scope available too.

So what I really wish was that languages used dot notation to go up scope eg ..name is that name two blocks out. This was one of the good parts of VB syntax which I miss in other languages. Think how much nicer it is than python's nonlocal and global, for example!

It's true that defer is more powerful at function scope. You can always recover block scope with an unnamed func. But it doesn't fit with normal lexically scoped constructs. You get gotchas.

I would love to be able to explicitly affect other scopes as you mention, for example to define two classes at the same time.

What's the use case for function scoped defer? I have never once needed function scoped RAII in C++ or any other language.

Collecting tasks from all iterations to await before returning.

Fair enough. But it's something like 5 lines of code to write that manually, and writing it explicitly is clearer. With implicit function-scoped defer, someone might refactor the code to inline the body of the function into its caller and break it.

I totally agree with you.

It's not terribly uncommon for larger functions to do something like

    func someSQLStuff() {
        tx, err := createTx()     
   
        defer func() {
            if err != nil {
                tx.Rollback()
                log(err)
            } else {
                tx.Commit()
            }
         }()

         rows, err = tx.QueryContext( ... )

         // more SQL
    }

Basically, function-scoped cleanup. Like closing opened files.

But that defer is already lexically at the function scope, so block-scoped defer would do the same thing.

Ah, sorry, you meant function-scoped vs block-scoped not just in general. Yeah, agreed.

I have a comment at a higher level with a broader example, but for Go at least this is somewhat common:

    func f(i interface{}) {
      if closable, ok := i.(closable); ok {
        defer closable.close()
      }
      // do stuff with i, maybe other casts, etc
    }

There aren't many nice options for "if I can call X, defer a call to X" aside from shoving it into an `if`, where it'd be captured by that scope. I mean, you could do something like

    deferrable := func(){}
    if closable {
      deferrable = closable.close
    }
    defer deferrable()

but imagine doing that every time. It'd work, sure, but it'd also be more annoying.

Couldn't you do:

    func closeIfNecessary(object interface{}) {
        closable, needsClosing := object.(closable)
        if needsClosing {
            closable.close()
        }
    }

And then just do:

    func f(i interface{}) {
        defer closeIfNecessary(i)
        ...
    }

Doing it this way also saves you boilerplate by factoring the downcast out into a separate function.

As long as you can always call .close() regardless of the code below, yep - that'd work, and is definitely more readable.

If you can't call it unless [some other conditions], it goes back to the same kind of problem though. "closable" may not be a good choice on my part, as they're often called unconditionally.

Both seem like fair options to me. With function scope, you can

    func f() {
      x := ...
      if x.something() {
        x.doSomethingEarlier()
        defer x.cleanup()
      }
      // use x however you like
    }

where scope-based forces you to do stuff like

    func f() {
      x := ...
      if x.something() {
        x.doSomethingEarlier()
        defer x.cleanup()
        // use x however you like
      } else {
        // use x however you like
      }
    }

In a scope-based defer, you'd have to keep all related code in the scope, nesting it another layer deeper / possibly duplicating it.

On the flip-side is of course that this doesn't work like most would probably want in function-scoped:

    for i := 0; i < 4; i++ {
      x := get(i)
      defer x.cleanup()
      x.whatever()
    }

and you're forced to

    for i := 0; i < 4; i++ {
      x := get(i)
      func() {
        defer x.cleanup()
        x.whatever()
      }()
    }

I've seen both of these patterns pretty frequently, in Go and in other languages. Go could, of course, have both a func_defer and a scope_defer, but that doesn't seem like it'd fit with the fairly strong focus on keeping the language feeling small and simple. So they had to pick one, and it can't handle both cases.

I think function scope is a good default, and wrapping in an anonymous function and calling it (like your last example) is a simple workaround to get the scope_defer behavior. If it was scope based there's nothing you could do to get func_defer behavior.

Yeah, I generally feel the same way. For fairly simple use, scope is more consistent (all scopes / closures are identical), but func is a bit more flexible if you're willing to pay with simple boilerplate.

I mean, you can convert them into each other. Scoped can do something like this (go+python blended code 'cuz lazy):

    func f(){
      deferred := []
      defer func() { for d in deferred.reverse(): d() }() // plus error handling
      if x.something() {
        deferred.push(func(){ cleanup() });
      }
      // same as func scope
    }

but that's a bit more ridiculous / error-prone (though a helper func is obviously possible) than the equivalent IIFE for func -> scope. More explicit, I suppose, but bleh.

It's more explicit, which is a good thing, as it makes the intent clear. This matters if, for example, the function is later refactored to inline into a caller.

It also allows more flexibility (do you execute them in the order they were enqueued, or in reverse?), more room for errors, confusion between different patterns / lack of consistency across different codebases, etc.

Explicit-all-the-things isn't an unambiguous Good Thing™. If it were, we wouldn't even be discussing this - it's an abstraction, which is less explicit than e.g. building defer out of a list and using GOTO.

> If it was scope based there's nothing you could do to get func_defer behavior.

That's clearly false. You could set up a list to hold objects to be disposed (or, more generally, closures to execute) and defer a simple procedure that disposes of all objects in the list. This is in fact what the implementation of defer must do internally.

I think by that he means it wouldn't be possible to get func_defer behavior in the example with a single keyword, or without some form of qualifier. The compiler wouldn't be able to differentiate the behavior.

I've never seen the first pattern in any other language. Aside from downcasts as in your other example, why would you only want to clean up an object if some condition is true? The 99% use case of defer is for resource destruction, which you nearly always want to do in the same scope the object was initialized in (and that observation in fact is what underlies RAII).

Reply on news.ycombinator.com