Hacker News Reader

We run a cluster of P2P, GPU heavy machines that use Go to ingest byte streams of raw radar data, store that info in btrees, and render, cache & serve map tiles that are drawn on the fly in response to http requests. We are not using much outside the stdlib (opengl and gdal bindings).

Garbage collection has become very fast in recent versions of the language. It's quite painless now, and we did struggle with it in the past. I'd take the current GC over dealing with other strategies unless you are a very specific, >60fps kind of situation. We literally don't think about the GC anymore.

Random stats from one of our busier nodes, over the past 20 min: 40GB currently in use, 391GB allocated (and freed) in the last 20min average of 2.6ms GC pauses every 70.6s 95th percentile on GC time is 7.26ms

For dependency management, there's a lot of tools that make things more automated and I think probably offer people what they are expecting/used to from other environments. This has been a sticky point for sure. I've settled on a bare bones solution that seems to work well: just using git submodules in the vendor directory within the project repo. Does everything I need (pin to version of a fork, build immediately after a clone and a submodule update --init). As of Go1.5, the vendor directory is supported by the compiler, and it will look there first to resolve dependencies. While this does work with recursive dependencies, you will want to flatten the dependency graph as much as possible if types are being used across multiple deps because types are distinguished by their full import paths. This encourages breaking things into modules so I'm ok with it.

First-tier support is a problem I have no doubt, but not one we've suffered from personally.

I'd say that debugging comes up a lot too, and Go doesn't have an official, reliable option for debugging. The Delve project is getting close to that though it looks like. I personally debug with the stdlib pprof package (provides an http server that prints stack traces for all goroutines currently active, allows cpu/memory/blocking profiles to be requested), print statements, general testing/experimentation and newrelic.

What company are you working at ? Looks like you do some interesting geospatial/map rendering stuff. Are you hiring ? :-)

How do you find the cgo overhead? Specifically with something like OpenGL, it seems like it would be a big pain point for rendering.

I don't have great measurements for this, but we have done optimizations to reduce the data flowing over the Go<->C interface. One of our key measurements that does make a big impact on performance is how often we need to upload data (that's not already over there) to the GPUs. So that's something we have worked on reducing (buffer reuse, compression). We also have a series of caches on the other side, so we aren't drawing more than we need to. It's hard for me to tease apart how much of these optimizations (and others) are ultimately aimed at addressing the cgo overhead, and how many are just typical stuff. The data we work with is cumbersome and my intuition is that there's probably a lot of room for optimization in our drawing even still, regardless of cgo. I wouldn't be surprised if a direct port C/C++ implementation of the rendering pipeline was significantly faster than ours in getting data into and out of the GPUs, but a big part of the project is data storage/networking/serving/caching as well and Go has bridged the gap for us (a small team that needs to build reasonably fast things reasonably quickly :)).

That's interesting. The cgo overhead was the only thing holding me back from considering it for games, since I didn't want to write a lot of C wrappers around the C libraries I want to use just to have them be more efficient, which is a shame, since Go is pretty nice, barring the C interop in some cases.

When I used to frequent gonuts, I raised the issue why they didn't went the FFI way as D, Rust, .NET, Delphi, FreePascal do, but sadly they rather use cgo as solution.

I'm pretty sure the problem is Go's use of segmented stacks and runtime, which makes it hard to properly interface with C efficiently.

Sure, but the compiler could generate that FFI code, why the need to rely on an external tool (cgo) and a C compiler?

FWIW I did measure the CGO call overhead a while back: https://github.com/slimsag/cgo-batching

Would the vendor management support deterministic builds?

Yes, that is the purpose of the vendor directory. You copy the source of each of your dependencies into vendor/ and commit it to your own source control so it can't change unless you do so yourself. I personally use git submodules to manage the contents of the vendor/ tree, but the go compiler/toolchain itself doesn't really care how you get the files into there.

Edit: other comments below have mentioned some of the tools available to manage your vendor/ tree. Using git submodules manually can be crude, especially when adding dependencies that themselves have other dependencies.

Yes. I check my vendor into git, so it's as deterministic as source code.

> Google itself doesn't actually have any real API libraries for their cloud services.

We're (I'm one of the contributors) working on them.

Check them out here: https://github.com/GoogleCloudPlatform/gcloud-golang

Datastore docs: http://godoc.org/google.golang.org/cloud/datastore

We've been running splice.com on Go for 3 years now and handle 5TB of audio/binary data per day. Our memory usage is around 10-15MB per server and the GC pause time has been really low. You do need to stream your IOs instead of reading everything in memory. In regards to dependency management, we honestly had no issues and now with vendoring, it's even easier. We do use a main repo with lots of smaller packages and only a few 3rd party dependencies that are vendored or available via private git repos. We don't use Google cloud but I heard they have 2 repos, one that has auto generated code and one that has hand written code (but less complete).

Could you elaborate on what you mean by stream your IO? Is this just using buffered io it is there a concept I could read up on here?

yes buffered io, you use readers to read/write small chunks at a time instead of loading everything at once. Go offers way to do both since in some cases, such as loading an entire file in memory can be fine/better/faster.

OT about splice.com: how do you version control DAW projects, you have a custom parser + text diff? Would you support Orion and Reaper? :)

Your tagline should be: splice.com - no more sample packs. Haha.

Nice business model.

> We continue to use Go because of its strengths, but it just really surprises me how little Google seems to care about the language and ecosystem.

Go is certainly a language that is used at Google, but AFAIK a lot of "Googlers" don't really like it and don't use it. It certainly not the "official language at Google", given the weight of C++ and Java there. But that's the consequence of being opinionated. Using Go means having Rob Pike over your shoulder telling you how to write code. And he made sure you can't escape that fact since there is no place for "ninja coding" with Go.

Google has some fairly specific style guides for all languages used: https://github.com/google/styleguide/blob/gh-pages/README.md

Go to me feels like they implemented the Google Java and C++ style guides as a language.

>> given the weight of C++ and Java there.

Python as well.

>> how little Google seems to care about the language and ecosystem

Let's compare with Microsoft. The top four out of five users at StackOverFlow have top tags in C#, I guess Google have a long way to Go.

Erm. StackOverflow was built by folk who used C# heavily, and cargo-culting is amazingly prevalent on both the C# and Java "traditional IT" communities, so _of course_ StackOverflow is biased towards those runtimes...

And yet the os group at ms is famous for their naked derision of the .net Framework.

Because of political wars that lead the .NET team to base the Framework on the CLR instead of the COM+ Runtime that was being prototyped.

Why do you think that WinRT is basically the return of .NET the Ext-VOS project origins, but with .NET metadata instead of COM type libraries?

.NET Native is what Ext-VOS was going to be, if it wasn't for the decision to create the CLR instead.

Hum .Net is their own dog food .. did you mean Android/iOS? :-D

What I mean was that M$ have competing offer to Android/iOS but it is not getting the upper hand even within their own organization.

Python isn't nearly at the level of C++ or Java there, at least not any more.

It is one of the official languages. Google is too big a company to have a single official language.

This slideshow clinches it for me. Go has some specific strengths that match low level, network infra types of problems. Beyond that, Go is not a good fit. This sounds like a criticism, but I don't think so. It's a compliment: it's a sharp tool for a specific kind of cutting. It's not trying to be some all-singing all-dancing language, which has gotten us all in to quite a bit of trouble.

That's exactly right, it looks like Go has found a niche of the network servers. Here is good overview by Andrei Alexandrescu: https://www.quora.com/Which-language-has-the-brightest-futur...

> but AFAIK a lot of "Googlers" don't really like it and don't use it

Where on earth did you hear that? It's simply not true.

(googler)

Wow. Is Google so homogeneous that you both can be so sure about your respective affirmations?

I must admit that I heard exactly the same from multiple Google engineers, who were all quite dismissive of it.

What is "ninja coding"? Sounds stupid.

I think it refers to the Perl-era idea that good code should be somehow "clever" rather than maintainable. It's how bad programmers who spend hours agonizing over how to reduce their line count (presumably to save disk space?) justify their behavior.

Personally, I think "cowboy coding" is the best derogatory name for this. IMO, what all of these ninjas have in common is that they don't realize that software development is a team exercise. Berkeley did a study on BSD and found that a file was opened 10x more often for reading that writing (i.e. people read code 10 times for every time they make a change). To my mind, "cowboy" conveys the proper amount of ignorance of the other people on your "team."

"Cowboy coding" already has a distinct meaning - it refers to writing code as fast as possible without concerns for technical debt.

The archetypal form of cowbody coding is the copy-and-paste: faster than any code reuse technique, but a booby trap for the future.

"Ninja coding", on the other side, refers to coding for cleverness' sake, at the cost of legibility and ease of use. No self-respecting ninja would simply copy-and-paste.

I totally agree with this sentiment. I think it's pretty well established at this point that in many cases clarity can trump optimization. Even more so when the optimization isn't performance based but rather line count based or "code golf" based. Also, do you happen to have a link to that berkley study? I can't seem to find it with a brief search.

Or, it is how some programmers continue to find motivation and fulfillment in the activity as an intellectual pursuit, yes at the cost of maintainability, but as opposed to being good little corporate drones writing a zillion dull lines of kindergarten-obvious code in Golang and Java that even the lowest decile of programming dunce can understand. Which is what Golang is designed to do. That's what they mean by optimization for large code bases: dumbing down. That's not a criticism by the way because that's optimal for large teams in large companies with large code bases and no room for too much creativity. But programming as art/fun, where appropriate, is also not (yet) completely to be dismissed as stupid, IMO.

For optimal code maintainability, there is a happy compromise between excessively terse and excessively verbose code. There is a lot of code in the wild that's too spread out.

Well sure, but erring on the side of verbosity at least ensures that someone can follow your thinking as long as the code is well structured. I would prefer to read 25 lines of decent Python over 3 lines of regexes in Perl.

Sure, but at least I would prefer 25 lines of decent Python over 125 lines of overly verbose Java. Nothing is black and white, it is easy to make code too verbose and thus very taxing for the reader.

Have a look at Elixir and Phoenix. GC is per-process so no global slowdown. Extremely fast response times and uptimes. Many other features that mesh nicely with webserving (some courtesy of Erlang's VM).

And a completely opposite philosophy from Go when it comes to error handling. Erlang/Elixir embraces failure (and immediately logs and restarts the process); Go seems to ignore it (unless you explicitly check, which to me seems... insane. For reasons why failing fast is better than failing silently, read https://blog.codinghorror.com/whats-worse-than-crashing/ .)

To any Go programmers ignoring errors is insane. I haven't seen code outside of simple examples ignore errors, and even there it's discouraged.

But you still have to check for errors after every line of code where they are possible (if you're covering all the bases), no? Due to lack of a traditional exception model?

Elixir/Erlang at least have pattern-matching which makes checks like that (from functions that return a non-OK value on errors) basically inlined:

`{:ok, val} = call_some_func(with_args)`

If it doesn't match on the :ok (i.e., an error occurred), you get a match error which gets logged, and the process typically gets killed and restarted by a supervisor process in a millisecond.

> Dependency management is a nightmare

Is it that bad now in 1.6 with vendor support? And a tool like `govendor` makes it easy to stick things inside of vendor.

> it just really surprises me how little Google seems to care about the language and ecosystem

To be blunt, google's priority is google, not the open source community or other companies using golang. Dependency management wasn't a priority because of mono repo. Having said that, they are pretty good about improving things for everyone, but it will never be like a company such as Typesafe whose product is the language and tooling itself.

is godep relevant still, or is 1.6 making something like govendor a "better" way to go?

I switched to Glide, a vendor-focused package manager, on 1.5. I think the community has already decided in favor of this approach. Post 1.5, Godeps' workflow just feels weird and unnecessary.

Use godep everywhere. It supports the vendor folder, and I find it's still a great solution (and saves me using submodules).

it is, and works well enough for individuals, but doesn't make for the friendliest ecosystem.

To be fair, Go's GC got A LOT better over the past few versions and it's getting much better still.

What I would really like to be able to do though is be able to write unmanaged blocks when I need them and know better than the compiler rather than having to "run my own heap" on top of a buffer like I typically have to do in managed languages that don't have that opt out.

Have you considered pausing the GC for those blocks? Or am I misunderstanding your use case?

How do you do this? I didn't know there was a way to temporarily disable the GC for specific blocks.

You can toggle the GC with https://golang.org/pkg/runtime/debug/#SetGCPercent. Turn it off before you enter your block and enable it again at the end of your block.

But can you guarantee that the GC actually makes progress that way?

Maybe you can, but maybe you shouldn't.

That is, you do this if you want a block to run without the GC interrupting it. That is, it's a really high-priority (realtime, or close) block. When it needs to run, it's the most important use of the CPU. But if you demand that the GC makes progress, you're saying that it also is important. And it is, but it's less important than the critical block. If you don't say that, then you wind up with "everything is important", and that way lies madness.

But you may be able to do it. You need enough CPU bandwidth that you can run your critical sections and spend enough time outside them that the garbage collector keeps up. (And then, every time you add to your code, you need to make sure that the CPU still has enough time...)

What does it mean for a GC to "make progress"? My understanding was that the OP wanted to stop the GC for a particular code block, so it would seem that any "progress making" would be undesirable.

https://golang.org/pkg/runtime/debug/#SetGCPercent

Do note that it triggers a GC when you call this, so it's not for all situations, but can be useful.

Crystal might be worth checking out if you have that kind of need.

> Google itself doesn't actually have any real API libraries for their cloud services. They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.

Are you sure? Google offers 2 Google Cloud APIs for Go:

- Google API Client Library for Go, which is auto-generated like you wrote;

- Google Cloud Platform Client Library for Go, which is intended to be idiomatically-designed.

See https://cloud.google.com/go/apis.

> They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.

Are idiomatic API libraries really a good thing? If an API is used heavily throughout your application, doesn't that mean it's time to wrap that part of the API in some idiomatic wrapper code that is meaningful in your domain?

Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like. And if the API call involves a web request, I am happy to just make the HTTP call myself. Half the time you need to understand the interchange format to debug your code anyway. My experience is that wrappers rarely protect you from that.

I guess if your application is 90% calls to an API, and you use a huge range of different functions in that API then you'd want a good, complete library. But how common is that?

You're arguing that it's good to (intentionally) break with common code idioms for certain classes of libraries??

Also, have you actually seen the auto-generated APIs he's referring to?

Here are the Godocs for the BigQuery API: https://godoc.org/google.golang.org/api/bigquery/v2

Here's some example code demonstrating the API's bad habits -- such as deeply nested struct pointers, and Do() methods everywhere: https://github.com/google/google-api-go-client/blob/master/e...

Here's the idiomatic library for bigquery: https://godoc.org/google.golang.org/cloud/bigquery

Thanks for that link, looks like that API was created after I started the project using BigQuery.

It's still labeled "experimental" though, which makes me a bit wary to use it just yet.

That's right. It's production-quality, but the API surface might change.

If you're OK with changing your code sometime in the future, then I'd recommend giving it a try for this or your next project. Changes will likely be minimal.

For a production app, wouldn't you "pin" your dependency to a particular version? I know I wouldn't be allowed to track master in my workplace.

Yes! Please do that.

Use something like git-vendor: https://github.com/brettlangdon/git-vendor

Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like.

Idiomatic APIs can be a real PITA, especially when you call them from a different language. KISS should reign supreme. (With bad 80's special effects in a made for TV movie filmed at a carnival.) If you leave it up to the client developer to build the idiomatic facade/interface, then the API is simpler, the code is more properly idiomatic in the end, and everyone is happier.

Just a quick note about dependency management: we use "go get" to download / update dependencies, integrate / test, and then just include them all in the project's git repo. Never had an issue.

Another major issue I have had with Go is database connectivity. The db drivers are really lackijg for Go. Makes it tough for those of us who use Teradata or Hive.

Well drivers are mostly written by database vendors or respective community in case of open source db. There is not much Go can do there.

You would like to write an entire server without using a GC?

I would not like to write a server with GC.

Source: I've written many servers in both C and Java and given the choice will never use a GC language again for a server. Holding out some hope for Rust..

There are definitely some pain points building a server in Rust, but I think on the whole the tradeoff is worth it, and will become moreso as the ecosystem matures.

Care to elaborate?

Presumably he is complaining about the GC implementation's characteristics, not its existence. You can read a description of the team's design decisions here: https://blog.golang.org/go15gc but a simple summary would be: memory is cheap and getting cheaper, so focus on making GC fast rather than small memory footprints.

Hasn't go been improving the GC with it's rounds of updates to the language?

Yes, with an emphasis on speed not memory.

actually thats what the Java G1GC (which will be the default in Java 9) does aswell.

However I would still favor java instead of go

And in my opinion, this is a great tradeoff. Ultimately, adding more RAM to a server (or clicking a button on the AWS console) is a very easy fix. However, reducing latency is almost never easy. If they can prevent GC stuttering, I'll take the higher memory overhead every time.

GC has gotten significantly better the last few releases, I don't think most applications will notice.

People do it all the time. For example, in C++ with modern owning pointers like std::unique_ptr<> and std::shared_ptr<> it's really not that bad.

> I tried a couple different server libraries on Python but they all seemed to struggle with even tiny loads. Not sure what was up with that

Could the problem have been the "global interpreter lock"? The GIL makes it impossible for multiple threads to simultaneously perform certain basic operations, effectively making the system single-threaded.

Invaluable perspective from the trenches--thanks.

Slide 13 (https://talks.golang.org/2016/applicative.slide#13) is interesting. I was expecting Go to be very close to C/C++ on the X axis (fast/efficient) as it doesn't use VM, but it is more close to Java ?

Ideally, Go should be pretty close to C, but currently there are reasons for it lagging behind (though there are also Go programs which perform better than their C counterparts):

- The Go compiler certainly isn't as sophisticated as the better C compilers. It looks that the code produces by Go 1.7 is quite improves vs. 1.6 with the SSA compiler and I would expect further gains in future releases.

- The Go compiler does not offer optimization levels and overall is optimized for compilation speed to, so these tradeoffs could be limiting.

- While the Go GC is very good and improving, so that the pauses are very low, it does use CPU, also while running in parallel to the program. How much exactly very much depends on the allocation behavior of the program, but should be accounted for.

It's true that Go compiles down to an executable and doesn't run on a VM like Java, but it does have it own runtime compiled into it, which manages the GC and goroutine scheduler to name two, so there is definitely some more overhead to Go then just straight C/C++

In addition, the language isn't designed for zero-cost abstractions like C++ is. You see this in the design of things like defer, which requires allocating records on the heap in certain cases, as compared to exceptions in C++ which can be implemented in a zero-cost manner.

I think interfaces are a better example, personally.

In my experience programming speed goes like this:

- 1x benchmark - C / static C++

- 2x slower - pure virtual C++ / objective-c

- 3x slower - statically typed GC languages (java,go)

- +6x slower - dynamically typed GC languages (js, python, etc)

After a well optimized implementation, speed comes down to manual vs GC memory management, static vs virtual function dispatch, dynamic vs static typing and heap vs stack allocation.

Compile speed is usually slowed down by any sort of type inference or templating/generics, which is probably one reason why go resists implementing generics or subclassing

That 6x is for when you have a JIT. Without one (I'm looking at you, CPython) it is 20-30x, with perhaps the exception of Lua.

https://benchmarksgame.alioth.debian.org/u64q/which-programs...

Two considerations...

Your rules of thumb are probably mostly related to CPU bound tasks. If database i/o is the bottleneck, then the differences across language probably don't matter much.

And, runtime performance isn't always the most important thing. In many cases, "time to market" or "cost of development" might be a higher order priority. In those cases, a higher level language might be more attractive.

Sorry, but I feel like I'm a little confused by your post. How can it be 6 times slower to program in JS or Python than C? Isn't this very much contrary to the conventional wisdom?

I think OP wrote programming speed but meant running speed.

Yup. "Speed of the programming language"

Im surprised to hear your issues with Python, even Django can handle decent load depending what youre asking of it.

dep mgmt, at least, is poised to improve a lot in the coming months. we've got growing consensus around some metadata files, and i'm nearly done with my SAT solver (github.com/sdboyer/vsolver), which will be in glide, and maybe others, soon.