Proposal: Less error-prone loop variable scoping
github.com
github.com
In C and C++, of course a for loop induction variable is reused for each iteration. And if you take the address of the variable and keep it around for too long, you get to keep all the pieces and possibly the exploitable UAF vulnerability, too. (Never mind that, prior to C99, C induction variables were always in an enclosing scope, and this would be fairly obvious from reading the code.)
In Rust, code like the examples wouldn’t compile. If you want to copy a value to the heap and take a reference, you need to say so. And it will be quite clear whether you are keeping a boxed copy or whether you are keeping a reference to the original object. (And the object can’t mutate out from under you unexpectedly in either case — a non-mutable reference prevents mutation!)
So the fact that, in Go, one might rely on automatic promotion of an induction variable (which obviously gets mutated every iteration!) to the heap, and thus get confused about precisely which value is promoted to the heap, seems weird from my perspective. IMO one might reasonably argue that the problem is that this pattern works at all, not that it works in the way that people usually don’t want.
edit: in C++, with range-based for, the induction variable is gone after each iteration. So taking a reference that outlives the iteration is invalid, and if you’re lucky, the compiler will warn. The confusing cases from the Go proposal are that invalid in C++, not confusing.
Loop variable scoping is not the only area where the designers of Go have failed to learn from past experience and instead opted for a design that lends itself to a more convenient implementation at the expense of exposing foot guns to the user: zero values are absolute dynamite, especially when combined with other language features like implicit zero initialisation of structs, multiple return values instead of sum types, or reflection-based magical JSON deserialization. When you have an empty string or a zero integer in Go, you can never be quite sure that the value you're holding is supposed to be what it is, rather than being an implicit zero value that snuck in at some point in place of a missing value.
Go also fails to offer any facilities to aid programmers in writing safe and correct concurrent code, beyond some fundamentally superficial language features like syntax extensions for channel types and spawning tasks. Channels are difficult to use correctly, usually having to be used in conjunction with some other synchronisation primitive like a WaitGroup, and compose poorly with implicit zero values resulting in the need to define semantics for fundamentally nonsense operations [1]. The language offers no facilities for restricting mutation, indeed in this respect, it's even worse than C. It's too easy to write buggy, racy code in Go.
There's no denying Go's initial allure, but in reality it fails to address in any systemic way the problems that have plagued the software development practice for decades. While other languages offer facilities for eliminating entire classes of bugs, all Go has to offer is an attitude that says "simply don't make mistakes." The whole language is a bitterly disappointing pile of missed opportunities. In the words of fasterthanlime, it's the billion dollar mistake all over again.
I find comments like this illustrate perfectly the gap between these focused on language theoretics, and those focused on building software.
Go is successful because it strikes a strong balance between what practical software engineering organizations need from a language, of which safety is only one constraint.
Languages also need to be: * Easy "enough" to learn * Easy to read * Easy to write * Performant * Compile quickly * Have strong tooling * Have a strong standard library * Provide good solutions for it's problem domains * Support large code bases with many dependents ... * And be "safe*
Easy here is also influenced by what people already know, syntax and basic working methods need to be familiar to the vast majority of software engineers.
Go is successful because for many engineers it provides enough of all of the above with *enough" safety.
P.s. complex, difficult to understand languages incur more bugs then simple ones, even with additional safety.
PPS: If you think multiple returns and sumtypes solve the same thing, you've not thought very hard on the subject...
Do you have any info on this? My understanding of this has been that some types of complexity make code worse, and other types make code better. For example, ruby's metaprogramming has been a source of bugs because of it's complexity.
On the other hand, haskell is "complex and difficult" in terms of having a powerful type system. In my personal experience, I've found that haskell code has far fewer bugs than go code, and a large fraction of the bugs I encounter in go would not have been written in haskell or rust, in a language with a more powerful type system.
Do you have any info on this link between "complexity" and bugs?
> Go is successful [...]
That's unrelated to whether it has systematic problems. As is much of your post to be honest. Like, yeah, javascript is successful. Bash is successful. No one will argue that those languages don't have systematic problems.
The parent poster wasn't saying no one uses go, but that people use go despite its systematic problems, and tbh your comment mostly doesn't argue against that thesis, just arguing that it's successful. Which sure, yes, it is, that's not really related to the parent comment's claim that it's a poorly designed language.
There have been some studies, and while all methodologies have tradeoff's go and clojure sit near the bottom of this list in bugs/committ: https://arxiv.org/pdf/1901.10220.pdf
Which is fascinating because they are very different languages whose unifying theme is simplicity.
But this actually fits well if you know another important fact: bugs occur along organizational lines. https://augustl.com/blog/2019/best_bug_predictor_is_organiza... If you've been around long enough, you realize a lot, if not most bugs, come from not understanding someone else's code. Type systems arn't going to help you, when you don't understand what the code they are "protecting" is doing.
> The parent poster wasn't saying no one uses go, but that people use go despite its systematic problems, and tbh your comment mostly doesn't argue against that thesis, just arguing that it's successful
But if go has so many problems why do people use it? Javascript, bash, and go have a unifying trait. They are useful, they solve a problem. But unlike javascript and bash, go wasn't and isn't integrated anywhere. No ecosystem forced people into go. They choose it, writ large. Why? Well because languages are more than there type systems.
You have linked a study which, if you read the abstract, effectively claims "we tried to reproduce a study about bugs-by-programming-language, and could not. We have concluded their methodology was flawed".
I can't find information in your linked source that support anything about type-system complexity resulting in more bugs.
> But if go has so many problems why do people use it?
If candy is bad for your health, why do people eat it? Surely people eating candy is proof it's good for your health. That's the argument you're making here.
We can show candy is bad for your health, and the comment you originally replied to provided a mix of reasons why they saw go as having problems and ignoring programming research. All of those are more evidence than what you've replied with, which is essentially "but go is popular", entirely ignoring their concrete points.
Anyway, if people choosing a language is proof that a language doesn't have problems, that surely means PHP, Java, python, C, etc have fewer problems than Go since they all are used much more than it.
Again, a language being good and people using it are not necessarily related. People eat candy despite it being bad for them. People code in Go despite it being designed like "C, but with GC"
Perhaps rob pike said it best:
> The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt
As rob pike so clearly said, perhaps go is successful because it is intentionally a poorly designed language, intentionally one that eschews programming research in order to appeal to python programmers.
As for the rest of your post, you dismiss the study as flawed. Fine. But most of the rest is your candy analogy, which is also flawed. One could just as easily say that people like you like... whatever languages you like... in the same way that people like cocaine, because it makes them feel smarter than everyone else. Such "proof by analogy" doesn't say anything meaningful or arguable.
The one other thing you said is that "the language being good" and "being used" are two different things. But I have a different view of programmers than you seem to. They aren't toddlers with no self-control reaching for what's bad for them (or drug addicts either, in my flawed analogy). Programmers use languages because, despite their flaws, the languages are still better for their use case than the alternatives. You have to judge a language as a whole package - syntax, semantics, build-in libraries, third-party libraries, ability to find people who know it, help available on Stack Overflow, all of it. Go (and PHP and Java and C/C++) have significant market share because, for significant classes of problems, they still are better than the available alternatives.
I trust the people actually building stuff. They are the hardest people to fool.
Right, bash and js have pretty significant problems, but they remain popular because they were developed as exclusive languages for platforms which became wildly popular; essentially they benefit from monopolies and network effects in a way that Go never has. The fact that Go is a popular language and still rapidly growing indicates that its issues are relatively minor compared with its advantages over other candidates (e.g., Haskell).
> On the other hand, haskell is "complex and difficult" in terms of having a powerful type system. In my personal experience, I've found that haskell code has far fewer bugs than go code, and a large fraction of the bugs I encounter in go would not have been written in haskell or rust, in a language with a more powerful type system.
Granted, but in my experience those languages trade off a lot of productivity in order to reduce bugs, and that leaves me a whole lot of time to debug Go programs and achieve a similar degree of correctness. Moreover, I often spend so much of my energy thinking about the type system in Rust/Haskell that I make silly mistakes that the type system doesn't catch (I've written a static site generator in Go and then in Rust, and I wrote a bunch of URL/filepath handling bugs in the Rust version that weren't present in the Go version--despite the benefit of hindsight--because I was focusing so much on the type system). Rust also makes it a lot more painful to create and use new integer types compared to Go (e.g., you have to define a ton of traits and even then you still have to wrap integer literals in MyIntegerType::new(0) and so on), so I see a lot of code that just passes around u64s as identifiers for different resource types and it's super easy to use the wrong u64 to index into the wrong resource collection.
I'm not saying that Rust results in more bugs on balance (I don't think it does) but that Go wins overall for most ordinary application code (but not for systems code, or high performance code, or real time / mission critical code).
> * Easy to read
The motivating examples of this whole proposal are examples where one reads the code and the semantics are unclear.
> PPS: If you think multiple returns and sumtypes solve the same thing, you've not thought very hard on the subject...
The stereotypical example of multiple returns in Go is functions that return val, err when they mean that they return one of a value or an error. Which is what sum types do.
The point is that with real sum types, the compiler will enforce that you return exactly one of a value or an error, whereas in Go, you're just supposed to do so, and it's possible to write code that mistakenly returns both or neither.
One of Go's core missions is to keep the number of concepts users need low. Multiple returns encompass the error use case and others.
But I would not personally design a new language without sum types or some other similarly strict solution for the kind of programming that will encounter errors.
There are many successful modern industrial languages out there that don't repeat the kinds of mistakes Go does. Kotlin is a strong example. So does the popularity of TypeScript show that we can successfully and fruitfully apply innovations from modern theory to industrial practise.
There's a tonne of real software being built in both of those languages. You don't need to ignore decades of research to build a successful and productive language.
Don't be so condescending to assume that Go's designers ignored decades of PL research. They knew it far better than you do.
And don't be condescending to the programmers, either. They aren't sheep who are led astray by shininess and PR. If that many programmers are using Go, then many of them find genuine value in Go. It solves enough actual problems well enough for it to get used, a lot.
This is a laughable claim when you've seen other languages (which have resulted in billion dollar companies with low engineering burden) that have done much better than go at this.
Go is successful for other reasons, mainly it plays into the biases of certain class of developers and managers.
While there is definitely an element of that, I think the main reason Go is successful are completely unrelated to the language itself. The main advantage of Go is that it is the one GC language with a slim runtime. Go programs are tiny compared to Java, C#, Python, Haskell etc. They start instantly. And they do so even if they are implementing an HTTP server or other non-trivial things.
So Go is excellent at something like containerized microservices, where you want to easily run dozens of containers on a single host, and rely on quick restarts for the occasional bug.
I have worked in places (only startups, no big tech) that deployed go (okay, low n=2) and this was not a consideration.
Eh, loop variable scoping has bitten me a single digit number of times in the eleven years I’ve been using it. Zero values are definitely more frustrating. Lack of Rust-like enums is another pain point.
But these are relatively minor compared to what Go gets right that other languages don’t—I would happily jump ship from Go the very moment something arrives that outperforms it for my use cases, and indeed I try out every language that I hear better-than-Go hype about.
So far Rust wins at systems programming, but even with the enums and lack of zero values I spend so much time fighting the language (not just the borrow checker) or implementing traits, etc—far more than I spend debugging nil pointers and such in Go. And frankly Rust is the only thing I’ve found that comes close. Go is just an extremely productive language even if it has a few sharp edges (after all, people still write really important code in C, C++, or even fully dynamic languages—compare that with zero values!).
Go ahead and ask them to remove this from the Frequently Asked Question section then.
This is weird for C++ programmers that think about the stack vs heap, but perfectly natural for Go. In Go, stack allocation for variables is a compiler optimization, not part of the language semantics. Go also only uses scope for visibility, not RAII.
That's fundamentally very weird from any systems background, and suggests a language design error: if people are returning pointers to "locals" because returning by value does an implicit copy, then maybe it shouldn't have been specified to do that.
That the runtime can allocate objects on the stack instead is an optimization, not a part of the language semantics. Letting the reference escape the enclosing function does not "automatically promote the object to the heap", it simply prevents the optimization above.
Also, none of this has anything to do with the problem of taking the address of loop variables. The confusing thing there is not the lifetime of the loop variable per se. It is the fact that the range-based loop variable is mutated to contain a copy of the next element in the range at the next iteration. It doesn't work like the C++ range-based loop, where the range_declaration is considered part of the loop body, for example.
Every other language I can think of that does this spells it in a materially different way, kind of like this (intentionally not matching any particular language):
x = func();
y = x;
This includes every GC language I’m familiar with except Go. It also includes C++ when using any sort of pointer (including smart pointers) and even plain references (with the obvious caveats about returning a reference to a local variable being nonsense in C++). It includes Swift with ARC. And it includes functional languages, pure and otherwise. If you return a (correct) reference to an object that would otherwise be local, you are returning the same type as the original local variable.
I think go is unusual in that it looks more like this:
x = func();
y = &x:
So I think woodruffw is reasonable to find returning the address of a local variable weird coming from pretty much any background.
However, similar things can be done in C# for example by capturing a local variable in a closure, even if that variable is of a value type (struct). The closure then becomes something similar to a Go reference to that value, and you are still free to return it with no UB or other problems.
Either way, if your mental model is adjusted to the GC memory model, there is nothing special going on here. The weirdness comes only out of tying an object's lifetime to its scope. Scope is irrelevant to lifetime in GC languages, where lifetime is tied to (live) references to that object. So, if a reference to an object "escapes" its scope, that object will live beyond the scope where it was originally declared - nothing strange about that.
Whether you agree or not, I definitely think that promoting loop variables to be heap allocated is consistent with the rest of Go.
(Obviously there are some consequences to this, wherein you may find yourself with heap-based allocation in hot paths that you didn't expect. But in terms of productivity, I think that not caring and having no distinction certainly reduces one thing that the programmer has to think about.)
I think there would be analogous design decisions that remove various low-level concepts in many languages. For example, Python ints are arbitrary precision, which removes a whole set of concerns from the programmer. Rather than being "weird", I think this is just optimizing for a particular kind of programmer productivity.
But this blog article satisfied me: https://ntietz.com/blog/rust-shadowing-idiomatic/
Automatic promotion is not that big of a factor, fundamentally this is a common issue with closures over mutable bindings, similar issues can be observed in other langages. Most (but not all) have mitigations already because it’s such a common issue especially around loops e.g. C# breaking-changed the semantics of range loops in 5.0, javascript added “let” and “const” which have block and per-iteration scoping, …
C, C++, and Rust don’t suffer from it (ish) because one doesn’t have closures and the other two require more explicit management of the capture and its lifetime.
C++ could have something similar with capture by reference but the code would probably not be straightforward, and the more straightforward code would have much bigger issues of dangling references.
For example, the entire problem leading to this proposal is about unexpected mutability. You have a variable i, which presently has a specific value. You take a reference to it, and you give that reference to some other code that saves it. Then you mutate i, and you are suddenly surprised that the external code now has a reference to the new value.
One can disagree with the wisdom of lexical closures, as opposed to other languages which require explicit declarations when binding variables or capturing values, but ultimately it's a tradeoff and lexical closures are a key architectural construct in Go. In this case, the implementation details resulted in poor cost/benefit, and tweaking the semantics of loop variables restores the cost/benefit in the context of Go's calculus.
In the OP, the examples involve closures that capture an induction variable. I don’t think Goroutines per se are involved. (Although, again, in C++ you had better not capture a reference and then have the closure in question outlive the referent, and you need to literally type & to get a closure to capture by reference. In Rust, references created by closure follow the same rules as any other reference.) And the earlier proposal:
https://github.com/golang/go/discussions/56010
gives an explicit reference (with an &) as its motivating example.
So I still believe that the whole issue being addressed is bizarre from the perspective of a C, C++ or Rust programmer.
(I’m having trouble thinking of another language that promotes a local variable to be a heap-allocated slot when one explicitly takes a reference to it. Most GC’d languages have local variables that are, themselves, references to the heap such that the referent (the heap slot / box / whatever you want to call it) will outlive the variable if the reference is copied. Pure languages may not strongly distinguish between a value and a reference to a value because the referent can’t change, so a copy is as good as the original. I think that Go is rather odd here, but I could easily be missing something.)
(P.S. C++ also has extends the lifetime of temporaries if a const reference is taken, but only to the end of the scope. This is too magical for my taste. At least the heap isn’t involved.)
I mean not really, a big reason for let and const in javascript was that this specific issue.
The worse thing is: the old code still compile in new /old version. There is no error or warning.
Updating the version is so innocent looking. Nobody will be able to catch this error.
Fortunately it was pretty obvious what was going on, but certainly the behaviour wasn’t what I expected while I was “in the flow”.
Super happy to see these little cases getting fixed!
The first part is good and will fix your problem. However the second part will produce some surprises to many people.
From what I can gather (I'm too busy/lazy/both to dive into the formalities), the for;; loop will have a new variable allocated at the start of each iteration, similar to the proposed range loop behaviour change. The strong argument for this is that you'd expect both loop forms to behave similarly; if they were to remain different, which would be the case of they didn't make this change, then the principle of least surprise would be violated.
However, reading the followup comments, some people seem to have missed that the value of the variable at the end of one iteration will be used to re-initialise the new variable at the start of the next iteration. This seems to mean that the usual semantics of a for;; loop will not change, other than if you're doing some really strange stuff, which doesn't seem to be the case when reviewing large existing code bases.
I had misgivings about the behavioural changes being triggered by the Go version number in go.mod, but rsc makes a good argument that any dependency change carries some risk, intentional or not.
I mean - maybe you feel differently, but if I saw that code in a PR, I would reject it. It’s way too tricky for me.
for v := range vs {}
for _, v := range vs {}
That would be a far more serious breaking change, however. Although, that would only in practice affect integer lists (i.e. code silently compiles). For other types, the compiler would complain about type mismatch errors and refuse to compile (which is better than nothing).
Needing either the index or both is likely a lot more common in Go than in other languages owing to the lack of any sort of iterator or composition. So e.g. any sort of zipping or in-place mapping requires the index.
> Although, that would only in practice affect integer lists (i.e. code silently compiles).
Not so. It would also affect strings (runes are integers), as well as maps with the same key and value types. Plus it could affect every other case depending how the item is used.
In our codebase, there's very few instances of needing the index. Usually you just want to iterate over a list and apply some function(s) to the elements or add them to another list.
Go indeed has
for v := range vs {}
For a change to be breaking it needs the potential to be able to break. As the potential approaches zero, so too does the coefficient of breakage.
for i, p := 0, (*int)(nil); p == nil; print(p == &i) {
p = &i
}
func foo(constFunc func(*[100000]int)) {
for a, i := [100000]int{}, 0; i < len(a); i++ {
constFunc(&a)
}
}The examples you share are in the proposal discussion, so there is little value in rehashing your apparently bad-faith argument here. It's not a breaking change if nothing breaks.
Go is and has always strived to be pragmatic. This is an incredibly pragmatic approach to solving a real problem. Being pedantic about this actually breaking a theoretical program which either makes no sense or doesn't exist does not add value to a proposal which already addresses this concept very well from the onset.
Obviously you can write it like that, my point is that it's not clear to me people actually are writing code like that. "If a footgun is never encountered then is it really a footgun?"