Go 1.7 toolchain improvements | Hacker News Reader

Go 1.7 toolchain improvements

dave.cheney.net

248 pointsspacey98 comments

If you're not a Go programmer, bear this in mind: the Go compiler got noticeably slower after 1.4, but it is still extraordinarily fast; "run debounced after every keystroke" fast; "rebuild the world on a whim" fast.

Personally, I think the Go community is a little unhealthily obsessed with this particular metric.

To me it's a broken window thing. Go's main mission is to make life easier for me as a developer, and compile times are a big factor in that. If we stopped being obsessed about it, we'd eventually end up with compile times reminiscent of C++.

exactly. I don't understand why gerrit's stop & frisk feature is so controversial.

Post 1.4 it went from "don't even think about it" to "mild annoyance".

Obviously nowhere near the pain of say C++, certainly something that was missed and will be nice to get back!

>Personally, I think the Go community is a little unhealthily obsessed with this particular metric.

Let's not forget that Go was invented to solve Google's problems, one of which is it took hours to compile their C,C++ codes. Compile time of Go 1.7, despite the improvement, is still 2x that of Go 1.4.

At it's slowest point, this compiler took single-digit seconds for its biggest workloads. So: I don't see how Google's C++ compile times are relevant.

C was already taking hours to compile in 2000, whereas languages with modules from Algol family would compile just in several minutes.

What I find positive about Go is making younger generations that only know C and C++ rediscover the compile times we had in the native compilation during the 90's.

Yes, in the mid nineties one of Delphi's killer features was super fast compiles. Partly possible due to a single pass compiler but it was just a very fast language to compile.

Really, most compiled languages except C++ are fast to compile. In the other thread I pointed out that Java compiles very fast, but nobody in the Go community likes to talk about that, it seems. Actually Java does have multi-tiered compilers, so the frontend is very fast as it doesn't optimise, and then the JIT compiler has a fast-but-low-quality compiler and a slower-but-higher-quality compiler, meaning you get the developer benefits of instant turnaround but the code still gets heavily optimised to GCC/LLVM quality in the hot spots.

I agree that the Java compiler itself runs fast, but many of the surrounding tools in the Java ecosystem are not fast. For example, Maven takes several minutes to rebuild Hadoop. Based on my experiences, neither Gradle nor ant are that much fast than Maven (although they are a bit faster.)

I disagree that "most compiled languages... are fast to compile." Scala is relatively slow to compile (I've seen Spark take over an hour to build). OCaml and SML are none too speedy to compile. Rust's compiler is rather slow (they're going to focus on speeding it up, I have heard.) It seems like most compiled languages that people actually use are slow to compile.

Go is a big exception due to its conscious decision to focus on fast compile times. This choice wasn't free-- it involved carefully considering tradeoffs. For example, Go compiles source code files unit-by-unit, forgoing global optimizations in the name of time. The grammar of the language is simpler, in order to enable more efficient parsing. Features like operator overloading are not present. It is trivial to tell what is a function call what is not-- there is no "syntactic sugar" like Ruby's function calls without parens.

Hadoop is 2 million lines of code. Yes, that's going to take a few minutes to compile no matter what language it's written in (if it has a compile stage at all).

By "most compiled languages" I guess I was thinking of the most popular ones, not literally all compiled languages. Sloppy phrasing, sorry. The top compiled languages according to TIOBE are Java, C, C++, C# (actually these are the top languages in general). Then they're all scripting languages until position 11 which is Delphi. So those are the top 5 and except for C/C++ they compile fast.

OCaml, SML, Haskell, Rust etc are all rather rare languages.

I don't agree that 2 million lines of code is "going to take a few minutes to compile no matter what [compiled] language it's written in." Based on my experiences with Go so far, I believe Go could get that significantly faster, probably under a minute. Also, the "several minutes" number I quoted is also for incremental compiles where only one very small thing has changed, not just for full compiles, which take even longer.

TIOBE is a poor measure of programming language popularity for a lot of reasons. But even if we accept that Java, C, C++, and C# are the top languages, I already commented that Java's build is not that fast in the real world (except for very small projects). I don't have any first-hand experience with C#, but I've never heard anyone claim fast compilation is an advantage of that language. C and C++ are slow builders, as we already know. So the top compiled languages are rather slow compilers, which is one reason why scripting languages (I assume you mean non-ahead-of-time-compiled-languages) became popular in the 2000s. This is one part of what Rob Pike was talking about when he said "poorly designed static type systems drive people to dynamic typing."

> TIOBE is a poor measure of programming language popularity for a lot of reasons.

And yet, it is completely consistent with pretty much all other measurements of programming language popularity (github, job postings, StackOverflow answers, etc...).

So maybe it's not that poor a measure after all.

According to TIOBE, Groovy rose from 0.33% to 1.8% in the last two months, and from 0.11% to 1.8% in the last 12 months. Click on "Groovy" from the main page and you'll get the graph at http://www.tiobe.com/tiobe_index?page=Groovy . Now say "maybe TIOBE's not that poor a measure of programming language popularity" with a straight face. These stats are gamed by language backers to make their language look good when selling consulting services and tickets to conferences, convincing programmers to contribute free work to their product, etc.

I would be skeptical of that, but I wasn't able to find LOC/sec stats for the Go compiler(s) anywhere. If you can point me to LOC/sec performance figures, that'd be interesting.

Gradle/Maven don't do incremental compilation at the level of the file, so yes, they're gonna redo the whole thing each time. Use an IDE for that. IntelliJ will happily do incremental compilation in Java projects at the level of the individual file (and there's no relinking overhead).

I understand that you're saying the top languages are slow builders, but it's just not the case. It sounds like you've used tools which simply aren't optimised for build times and judging the language based on that. But I usually have roundtrip times between editing my code and running the recompiled program of about a second when working with Java.

I have used both Eclipse and Intellij for building Hadoop, and neither could get incremental compilation under 30 seconds. In fact, it was often much more than that. Eclipse was a huge pain to use with Hadoop because it was always getting something wrong-- many times when you updated the project dependencies, Eclipse would essentially break and need manual fixing. Even loading up the project in either IDE takes at least a solid minute, with an SSD, Intel i7, and 16 GB of RAM.

Prior to using Java, I used C++ for 10 years. Every project I worked on had compile time problems. It was an open secret that we spent most of our time waiting for compiles.

It's very common for people working on small projects to feel good about their compile times. But sometimes projects grow up, and they outgrow the language and the tools they were designed in.

TIL that there are still people who believe that parsing has any impact on compilation speed.

Do you know Walter Bright?

The guy that created the first C++ native compiler and one of the D authors?

http://www.drdobbs.com/cpp/c-compilation-speed/228701711

> Let's not forget that Go was invented to solve Google's problems

No, it was invented to solve the problem that a handful of Google engineers experienced while writing C++ at Google.

Most of Google's code base is written in Java and as such, doesn't suffer from slow compilations.

I suspect there's a bias born of some of the key go folks working in the context of truly leviathan single repos. It shows up in the slowness to understand what other people need re package management as well.

It just strikes me as amusing, since the tremendous amount of whining about how much of compilation time (compile-time) generics would add makes the issue appear important, yet here we have a 2-10x compilation time slowdown, but in this case it's suddenly totally fine -- the Go community is obsessed about the wrong things.

There are still many things to optimize. One of the reasons to rewrite the compiler in Go was to make it easier to profile. The generics issue considers best-case compiler performance.

Why would being written in Go make anything easier to profile?

by being able to use the go toolchain on the go source itself (it was previously in C)

but in this case it's suddenly totally fine

Not at all - it was one step back while getting on the right path, then speeding up again.

I'm not sure how repeatedly apologizing for the slower compile time, promising that it's going to get better again in the future, putting an entirely new backend on the compiler, and making it a big focus of every single post about the last three Go versions constitutes being "suddenly totally fine".

To be hypocritical, there must be a difference between words and actions; it looks to me like "we're really concerned about compile times" has been backed up by actions, even if there was a regression. I don't recall them ever promising that there would never be a compile time regression because it was their #1 concern; what I recall them promising as their #1 concern was almost-but-not-quite total backwards compatibility, which they've maintained.

This is a temporary regression. I guess generics would be a permanent slow-down, and might not scale nicely for larger projects - although I do now know enough about the workings of compiler algorithms to judge that.

I don't think generics would result in a compile time slowdown relative to the alternatives. If you don't have generics, programmers work around it with code duplication or they work around it with dynamic typing (interface{}, but also the runtime support for maps and channels fall into this category). Not coincidentally, these are the two main ways to implement generics in a compiler.

> I think the Go community is a little unhealthily obsessed with this particular metric.

I've been on teams that are not obsessed with this metric and what very often happens is that every patch landed makes the system just a tiny bit slower. The next thing you know, your tools feel glacial and users are miserable.

And, in fact, at work right now my team is trying to feverishly claw back some performance we lost over the past few months without really noticing.

If you're working on a system where performance is a key part of the UX, you really do have to be vigilant about it.

Go's dependence on source level transformations to address gaps in the language make it a reasonable requirement.

There's something vicariously gratifying about posts like this. Optimization is some of the most fun programming to do: you have a crystal clear objective goal to go for and once you've got a profiler set up it's just:

1. Find slow thing.

2. Make faster.

3. See benchmark improve.

Some of the most satisfying coding I've done has been optimization.

I agree with this, but there's one more aspect: you learn something. Deeply. "Oh, I didn't realize that would do _that_..."

It's also taught me just how much we can guess incorrectly the first time. It's a skill you can get better at. "Profile first" is a good heuristic, but "guess first, profile second, _then_ do the work" means you train yourself too.

I'm still very much a noob at this.

I think every time I've used a profiler, it's taught me that me intuition of where the bottleneck was was completely wrong. It's a humbling experience, in a good way.

Well, that's only satisfying when your other thing doesn't get slower/worse because of that. And then turns out you're out of low-hanging optimization fruit and you have to choose trade offs.

I'm glad to see a return to improving compiling speed and binary sizes. Thank you to the Go team!

- faster compilation times

- smaller binaries

good job.

Faster compared to 1.6, still much slower than 1.4.

C is still the king of speed.

Not when compared with languages that have modules.

In 2000 I worked on a project where "make clean all" would take 1 hour per platform/build type.

Oh I meant in execution, not build time.

Similarly, I remember the days when junior Assembly programmers were able to outperform C compilers for home computers.

You can find interesting details to this post in this thread:

https://groups.google.com/forum/#!topic/golang-dev/DpMyPbclI...

I'm confused how Go can be paraded as a low level systems language, which improves performance via concurrency, and yet the graph says it's compiler is still over twice as slow as when it was written in C?

Its an irritating dual meaning of "system's language". For Rust, D, C++ and the rest of the world, "systems" implies drivers, OS and embedded systems where low level memory management is key. For the go developers, a system is a web-service so its actually competing with GC languages like node.js and Ruby rather than C.

But the D standard library also relies largely on a garbage collector, so it's questionable that it differs much from Go in that respect.

As most things, it's a spectrum and not binary. Userland is also part of an operating system and could be written in Go without much trouble. One could also argue that things like Docker are also systems programming.

Go is less 'systems programming' than C, C++, or Rust, because it has a garbage collector. Go is more 'systems programming' than Java, C#, node.js or Ruby, because it compiles to native binaries, provides value types (C# does too), integrates more tightly in standard UNIX APIs, etc.

FWIU D is actively trying to reduce it's dependence on a GC, while go has decided that the GC is here to stay.

BTW, as D can call C functions. Thus, in theory, you could choose to use only the C standard library, and use D as a "C with a better syntax".

This way, no garbage collector nor any bloat gets linked into your program ... but you still benefit from RAII, templates, array slices, and simplified declaration syntax.

You can write an operating system in a garbage collected language. One example, Oberon System:

https://en.wikipedia.org/wiki/Oberon_(operating_system)

I don't know anyone who has written more than a toy OS in Go, but it does not seem a fundamentally impossible task. Getting started is harder, because the earliest parts of the kernel would be modifications to the runtime package. Once you get that out of the way, writing drivers could be quite fun.

But to the broader topic, I don't have a good definition of "systems language" so I'm not going to claim Go is one.

(I work on Go.)

An OS often has to generate nearly every opcode on the target machine (many are there just for this purpose). Can Go generate arbitrary ASM instructions?

An OS has to deal with often strange calling conventions (interrupt routines, OS entry point traps, multiple language support). Can Go express the calling stack in any way?

Drivers are often interested in controlling timing and latency down to the instruction. Can generated Go code be examined by the developer for instruction counts?

A good kernel language is pretty low-level. Used to write them all in assembler.

Go has an assembler: https://golang.org/doc/asm

I have used it in sticky situations, like the trampoline for a signal handler, and writing functions that have to conform to a C ABI because they are called by the OS. (In particular, a loader .init function.)

To minimize how much assembly a theoretical OS author has to write, there are a couple of extra tricks in the runtime package they could borrow, like //go:noescape annotations. You have to be careful when writing code like that, but it does the job.

Before the Go runtime was rewritten in Go, I worried that for a task like writing a kernel there would be a lot of assembly. But I have been pleasantly surprised by how little is needed in the runtime, and that gives me some hope for a project like that.

Of course this is speculation on my part, I could be wrong.

Hi crawshaw, thanks for your work on Go in general and smaller binary sizes in particular. Could you explain why cmd/go generated by Go tip remains larger than the one generated by Go 1.4.3?

My guess is that Dave Cheney is measuring the cmd/go that comes with the release, meaning it's a different program each time it is measured. Someone probably snuck some new features into it between releases. :-) Like all the -buildmode support for shared libraries, building .so files to load from C programs, etc.

I have been using it in my binary size measurements as I work on http://golang.org/issue/6853, and it has shrunk reasonably well since the 1.6 release. More to come!

Even though I am not a big fan of some of Go's design decisions, this is one area where I do say that Go is a real systems programming language specially since 1.6, regardless of what C guys say.

You have OSes written in Lisp, Cedar, Modula-2+, Modula-3, Oberon, Oberon-2, Active Oberon, Component Pascal, Sing#, System C#,...

All systems programming languages which do have a GC for heap management.

Given that the Go compiler and runtime are written in Go, with the necessary Assembly glue code, I would say Go is no less systems programming language than those referred above.

Even ANSI C, cannot be fully implemented without Assembly or compiler specific extensions for hardware integration.

But of course the naysayers will only change their discourse the day someone decides to invest their time and produce a bare metal runtime for Go, following the approach of one of those OSes that I mentioned.

It's not a low-level systems language in traditional view of what "system" is. For Google, "system" is a cloud, not the hardware, so take it from this perspective.

By the way, here is a nice video of Q&A session with Alexandresku, Matsakis, Stroustrup and Pike (D, Rust, C++ and Go creators), where they also explain what does 'systems language' mean.

I would like to see that video too...

here https://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2014/Pa...

Think it's this https://m.youtube.com/watch?v=BBbv1ej0fFo

I think you refer to this video: https://youtu.be/BBbv1ej0fFo?t=3m40s

(bookmarked at the start of the talk about system programming)

This one? https://www.youtube.com/watch?v=BBbv1ej0fFo

Yep!

Alexandrescu has probably done more to popularise D than anyone else, but Walter Bright holds the "creator" bit.

It's worth noting that in Rust terms, this video is a bit old. I haven't watched it in a while, but some stuff might be inaccurate now. For example, Rust still had a runtime and green threads then.

who's parading it as that? Certainly not the Go team

https://talks.golang.org/2012/splash.article

It was introduced with those words. They have changed since (a long time ago too).

At the very top of the Go web site it used to prominently say "a systems programming language" [1].

[1] https://web.archive.org/web/20091112094121/http://golang.org...

"It was introduced with those words. They have changed since (a long time ago too)."

Yes, I read that comment. I was providing evidence to back up that claim, in case you did not notice.

Thanks for doing that

I don't think the Golang people are parading it as a "systems language". Part of the goal was to make something fast that's both safer and easier than C. I think they succeeded with that. But I don't think anyone expects a language with enforced safety and built-in garbage collection to be faster than C. That just isn't going to be possible.

> Part of the goal was to make something fast that's both safer and easier than C.

Right, and also fast compilation.

I remember reading the goal was ~20% slower than C for overall performance.

It was auto-translated from C to Go. This means, they replaced well-written C code with badly (unoptimized, nonidiomatic) Go code. The reason was, to guarantee that the initial Go based version was identical to the proven C version.

With the releases subsequent to 1.5 the cleanup and enhancement process of the Go version started and now it shows nice results.

I think its one really great feature of Go, that the whole build/runtime infrastructure is written in Go itself. So, everyone who is fluent in Go can easily read and possible enhance the infrastructure.

big question & fair point.

If the system includes the task of its own maintenance (and therefore the developers who work on it), go is faster than C because of maintenance and correctness advantages. (that definition of system is more like 'systems biology' than 'close to the iron software that calls OS APIs').

If golang stays popular the compiler will keep getting faster. Especially because now that it's self-hosting, future improvements will improve compile & runtime perf.

As someone new to the joys of golang, it is hard to imagine faster compile time or smaller binaries. It just keeps getting better.

it is hard to imagine ... smaller binaries

I like Go a lot, and I don't fret about binary size. But let's not be too charitable :). The CLI apps I write usually start trending towards 10MB with a few package imports. Their equivalents in C are usually less than 100k. One can imagine getting to better than a 100x difference.

Go binaries are static, with no external dependencies. You can't win that battle.

> it is hard to imagine ... smaller binaries

> I like Go a lot, and I don't fret about binary size. But let's not be too charitable :). The CLI apps I write usually start trending towards 10MB with a few package imports. Their equivalents in C are usually less than 100k. One can imagine getting to better than a 100x difference.

Who want smaller binaries can use this compressor tool: http://upx.sourceforge.net/ It may be a bit slow but you just need to use it in the distributed version.

That only saves disk space, though, not memory.

Well.. the C binaries depend on libc on your system and come with no type safety and automatic garbage collection. Go binaries are fully static (except for networking typically and you can switch that to static as well).

What happened between Go 1.4.3 and 1.5 which introduced such a slow down in compile times?

They rewrote the compiler from C to Go.

Well, at least I bet the compiler compiles itself faster now :-)

To be specific I think they used a transpiler, probably Russ Cox's c2go (https://github.com/rsc/c2go).

What's the point of doing that? I get that you want a language to bootstrap itself, but if you skip the experience of actually writing the compiler, and the code you have to maintain is a weird non-idiomatic machine-generated thing, what have you gained?

Automatic translation for most of the code where it doesn't make a difference

Then you can get this base and start improving on it

More developers (who don't know c (good enough), but are proficient in go) who can contribute to the compiler code long term (and improve the transpiled compiler). Other reasons would interest me as well.

Are there large numbers of people skilled in writing compilers, but don't know C? I doubt it.

That practical point aside, I do think it's good to lower the barrier of contributing.

I think it is similar to what some people say that they will contribute to open source project only if it is on github. So it is not logical but nonetheless many want it.

Yes, I doubt it too. But are they interested in contributing to the go compiler if they don't already do that? I would guess go users are more interested in contributing to go. Plus the compiler people who are good at c and are already contributing can probably get up to speed with go quickly if they want to and don't know go yet. Or they already know go because they use it.

from https://talks.golang.org/2015/gogo.slide#4

  "Why move the compiler to Go?

  Not for validation; we have more pragmatic motives:
  Go is easier to write (correctly) than C.
  Go is easier to debug than C (even absent a debugger).
  Go is the only language you'd need to know; encourages contributions.
  Go has better modularity, tooling, testing, profiling, ...
  Go makes parallel execution trivial.
  Already seeing benefits, and it's early yet.
  Design document: golang.org/s/go13compiler"

and https://talks.golang.org/2015/gogo.slide#10

  "Why translate it, not write it from scratch? 
  Correctness, testing.
  Steps:
  Write a custom translator from C to Go.
  Run the translator, iterate until success.
  Measure success by bit-identical output.
  Clean up the code by hand and by machine.
  Turn it from C-in-Go to idiomatic Go (still happening)."

As an avid go user, I'm excited about these improvements, especially the garbage colleciton

Can someone tell me why they went from C to Go despite the huge difference in compilation time?

Long story short it's easier to code in Go than C. Here is a more complete explanation https://talks.golang.org/2014/c2go.slide#1.

https://golang.org/s/go13compiler

You refactor before you improve

The only thing preventing me from using Go on Windows is lack of production quality cgo on Windows x86 and x64.

For example, using external linking should "just work" with recent versions of sqlite3 but it fails on Windows.

I had no problems building go-sqlite3[1] on Windows 7 x64 using the toolchain provided by the Win-Builds project[2]. It's using gcc 4.8 and sqlite 3.10, so maybe that doesn't hit your meaning of "recent versions". It was certainly recent and easy enough for my needs :)

[1] https://github.com/mattn/go-sqlite3

[2] http://win-builds.org/

I tried TDM-GCC.

In my experience, certain aspects increase risks when using go:

1. go is buggier for Windows than Linux and FreeBSD, even though they are all "first class" platforms in go.

2. cgo is also buggier for Windows than Linux and FreeBSD. cgo is not go, as Rob Pike said, so this is listed separately.

3. 386 was buggier than x64, at least in go 1.2 to 1.3 on Windows. I wish I could ignore 386 and XP but I cannot.

So if a project includes all of the above aspects, well...it wasn't a good outcome for me, so I ended up sticking with C and C++ on Windows for non-hobby software.

However, I've been very happy with go 1.4.3 + cgo + x64 on FreeBSD 10 and will probably update to go 1.6 or 1.7 within a year.

Regarding go on Windows, I'm hoping to see go-sqlite3 issue #272 resolved: https://github.com/mattn/go-sqlite3/issues/272

And go issue #14397 cmd/link, runtime: panic "invalid spdelta" with -ldflags="-linkmode internal"

And go issue #10776 cmd/link: windows cgo executables missing some DWARF information

And go issue #12516 runtime: throw in linked c++ library causes app crash even if caught, on windows platform

And so on...

I really love using go (with cgo) on non-Windows platforms, but I can't see this combo being useful on Windows for mission critical projects for a while. There's so much more to go than the language itself and switching gears to use other languages that don't offer that is painful.

Maybe LXSS.sys will eventually make this irrelevant -- at least for companies that migrate from older versions to Windows 10 or to a non-Windows OS.

Works fine for me, but I had to install the x64 version of MinGW

(I went to high school with Dave. Hi, Dave! Signed, misc. Banyule Alumnus)

Reply on news.ycombinator.com