Hacker News Reader

Comment by kubb | Hacker News Reader

Russ Cox and the Go team learned that the loop variable capture semantics are flawed not by reflecting about how their language works, but through user feedback.

This could have been prevented by having one person on the team with actual language design experience, who could point this issue out in the design process.

In this case, after 10 or so years, and thousands of production bugs, they backpedaled. How many other badly designed features exist in the language, and are simply not being acknowledged?

If you point it out, and you're right, will you be heard if you don't have a flashy metric to point to, like a billion dollars lost?

What if the flaw is more subtle, and explaining why it's bad is harder than in this very simple case, that can be illustrated with 5 lines of code? What if the link between it and its consequences isn't that clear, but the consequences are just as grave? Will it ever get fixed?

View on HN

Others have suggested that Rob Pike and Ken Thompson have some language design experience, to state it mildly. I also want to point out...

> Russ Cox and the Go team learned that the loop variable capture semantics are flawed not by reflecting about how their language works, but through user feedback.

I think "user feedback" isn't the whole story. It's not just the Go team passively listening as users point out obvious flaws. I've noticed in other changes (e.g. the monotonic time change [1]) the Go team has done a pretty disciplined study of user code in Google's monorepo and on github. That's mentioned in this case too. This is a good practice, not evidence of failure.

[1] https://go.googlesource.com/proposal/+/master/design/12914-m...

> Rob Pike and Ken Thompson

They are huge names in the field, but honestly, they just suck at language design itself.

Can you substantiate that or is this just flaming?

I'm sure they could come up with a list of language decisions they disagree with.

I'm equally sure that if you asked kubb, kaba0, and three other strongly opinionated folks for a list of good language designers, each of the <5 lists you get back would be very short, and there'd be no overlap between them.

Many languages have made this mistake, despite having engineers and teams with many decades or centuries of total experience working on programming languages. Almost all languages have the loop variable semantics Go chose: C/C++, Java, C# (until 5.0), JavaScript (when using `var`), Python. Honestly: are there any C-like, imperative languages with for loops, that _don't_ behave like this?

That decision only becomes painful when capturing variables by reference becomes cheap and common; that is, when languages introduce lightweight closures (aka lambdas, anonymous functions, ...). Then the semantics of a for loop subtly change. Language designers have frequently implemented lightweight closures before realizing the risk, and then must make a difficult choice of whether to take a painful breaking change.

The Go team can be persuaded, it's just a tall order. And give them credit where credit is due: this is genuinely a significant, breaking change. It's the right change, but it's not an easy decision to make more than a decade into a language's usage.

That said, there may be a kernel of truth to what you're alluding to: that the Go team can be hard to persuade and has taken some principled (I would argue, wrong) positions. I'm tracking several Go bugs myself where I believe the Go standard library behaves incorrectly. But I don't think this situation is the right one to make this argument.

This isn't a bug in java. Java has the idea of "effectively final" variables, and only final or effectively final values are allowed to be passed into lambdas seemingly to avoid this specific defect. Ironically, I just had a review the other day that touched on this go "interaction".

The outcome of this go code in java would be as you'd expect, each lambda generated uses a unique copy of the loop reference value.

Oh, today I learned. I think this was an issue in Scala (with `var`), but this seems like a great compromise for Java core.

I suppose Java had many years after C#'s introduction of closures to reflect on what went well and what did not. Go, created in 2007, predates both languages having lightweight closures. Not surprising that they made the decision they did.

Your comment inspired me to ask what Rust does in this situation, but of course, they've opted for both a different "for" loop construct, but even if they hadn't, the borrow checker enforces a similar requirement as Java's effectively final lambda limitation.

Newcomers to Java usually dislike the "Variable used in lambda expression should be final or effectively final" compiler error, but once you understand why that restriction is in place and what happens in other languages when there's no such restriction, you start to love the subtle genius in how Java did it this way.

Go, designed between 2007 and 2009, certainly had the opportunity to look at their introduction in C# 2.0, released 2005, or its syntactic sugar added in C# 3.0, released 2007.

I think that's an ahistorical reading of events. They did have the opportunity, but there were very few languages doing what Go was at the time it was designed. My recollection of the C# 3 to 5 and .NET 3 to 4.5 is a bit muddled, but it looks like the spec supports a different reading:

C# 3.0 in 2007 introduced arrow syntax. I believe this was primarily to support LINQ, and so users were typically creating closures as arguments to IEnumerable methods, not in a loop.

C# 4.0 in 2010 introduced Task<T> (by virtue of .NET 4), and with this it became much more likely users would create a closure in a loop. That's how users would add tasks to the task pool, after all, from a for loop.

C# 5.0 in 2012 fixes loop variable behavior.

I think the thesis I have is sound: language designers did not predict how loops and lightweight closures would interact to create error-prone code until (by and large) users encountered these issues.

This bug appears to be because Go captures loop variables by reference, but C++ captures are by copy[1] unless user explicitly asked for reference (`&variable`). It seems like the same bug would be visually more obvious in C++.

[1] https://eel.is/c++draft/expr.prim.lambda.capture#10

> JavaScript (unless using `for...of`)

The change in Javscript doesn’t have anything to do with for…of, it’s the difference between `var` and `let`. And JS made the decision to move to `let` because the semantics made more sense before Go was even created (although code and browsers didn’t update for another several years). That’s why Go is being held to a higher standard, because it’s 10+ years newer than the other languages you mentioned.

Ah, thanks for the correction! That's right. But:

1. Did JS introduce this change after Go's creation? Yes. (And also after C#.)

2. Did arrow functions support precede let and const support? Yes.

Answering the second question and finding the versions with support and their release dates answers the first question.

             Arrow functions    let and const
    Firefox    22 (2013)          44 (2016)
    Chrome     45 (2015)          49 (2016)
    Node.js     4 (2015)           6 (2016)
    Safari     10 (2016)          10 (2016)

This places it nearly 10 years after the creation of Go. And with the exception of Safari, arrow functions were available for months to years prior to let and const.

This is somewhat weak evidence for the thesis though; these features were part of the same specification (ES6/ES2015), but to understand the origin of "let" we also need to look at the proliferation of alternative languages such as Coffeescript. A fuller history of the JavaScript feature, and maybe some of the TC39 meeting minutes, might help us understand the order of operations here.

(I'd be remiss not to observe that this is almost an accident of "let" as well, there's no intrinsic reason it must behave like this in a loop, and some browsers chose to make "var" behave like "let". Let and const were originally introduced, I believe, to implement lexical scoping, not to change loop variable hoisting.)

C# made the mistake not when they introduced loops, but when they introduced closures, and it didn't become evident until other features came along that propelled adoption of closures. Go had closures from the beginning and they were always central to the language design. C# fixed that mistake before the 1.0 release of Go. But the Go team didn't learn from it.

I hate to be that guy but this would not be possible with rust, as the captured reference could not escape the loop scope. Either copy the value, or get yelled at the lifetime of the reference.

This is one of the things the language was designed to fix, by people that looked at the past 50 years or so of programming languages, and decided to fix the sorest pain points.

> JavaScript (when using `var`)

I would argue that var is an entirely different issue. If variables last the entire function then it's far less confusing to see closures using the final value. After exiting the loop the final value is right there, still assigned to the variable. You can print it directly with no closures needed.

Are you suggesting Rob Pike has no experience designing a programming language?

I think the parent was trying to imply that Ken Thompson had no experience in designing a programming language :-)

Seriously though, "having experience" and "getting things right" are two different things, although Golang got a lot of things right, and the parent is being unnecessarily harsh.

Or Ken Thompson?

I should say language design knowledge.

Are you aware who these people are?

I can’t find it now, but I remember some joke about “it’s an interesting language, but why did you ignore the last 50 years of programming language design?”

I find Go quite frustrating in how it decries how over-complicated some features are, and slowly comes around to realize that oh, maybe people designed them for a reason (who woulda thunk it?).

I answered some of this above: https://news.ycombinator.com/item?id=37577312

> This could have been prevented by having one person on the team with actual language design experience

I thought you were serious, right up to that bit. Well played. I hope.

> This could have been prevented by having one person on the team with actual language design experience, who could point this issue out in the design process.

Instead of making a mistake, they could have simply not.

See also RFC 9225: Software Defects Considered Harmful https://www.rfc-editor.org/rfc/rfc9225.html

> Russ Cox and the Go team learned that the loop variable capture semantics are flawed not by reflecting about how their language works, but through user feedback.

Since "Go 1" was deemed complete and the "Go 2" project began in 2018, the direction of the language was given to the community. It is no longer the Go team's place to shove whatever they think into the language. It has to be what the community wants, and that feedback showed it is what they want.

The Go team never shoved anything into the language without good reason, and they will not allow the community to shove anything into the language; that's how we got half baked classes in Javascript and half baked functional programming in Java, or the overall trend of languages taking features from other languages because community members say "this language would be better if it had features from this other language" often enough.

The "Go 1" project was centred around the Go Team. They built what they wanted and needed with little regard for the rest of the world. If they felt loop variable capturing was important, they would have added it. Of course, they didn't find it necessary, so it wasn't added.

When "Go 1" reached its natural stopping point and closed down, the "Go 2" project emerged to continue development of Go under the wants and needs of the community. That capture is being added now because the community has shown a desire to have it. The Go Team may use their expertise to guide the community in the right direction, but we are here because the "Go 2" project is community driven.

The original commenter seems unaware that the project changed hands.

> How many other badly designed features exist in the language, and are simply not being acknowledged?

Very few.

> If you point it out, and you're right, will you be heard if you don't have a flashy metric to point to, like a billion dollars lost?

If you're right yet don't have a better idea then what do you expect to occur?

> What if the link between it and its consequences isn't that clear, but the consequences are just as grave?

The consequence is your developers must be careful with loop variables or they will introduce bugs. That's not particularly "grave" nor even especially novel.

I'll admit, it's not a good ivory tower language, but then again, that's probably why I use it so often. It gets the job done and it doesn't waste my time with useless hypothetical features.

> If you point it out, and you're right, will you be heard if you don't have a flashy metric to point to, like a billion dollars lost?

Is this a subtle nod to the billion dollar mistake?

Because they deliberately included the billion dollar mistake as part of the language.

Even if they knew better than to include the billion dollar mistake, they were probably aware that they couldn't make a popular language without including it.

Because most users demand it.

So whats your point ?, old ideas never die ?, language design is not language purpose and goal ?, they made a mistake creating Go ?, refusing to find something suitable or just break compatibility?

Reply on news.ycombinator.com