Go Execution Modes | Hacker News Reader

Go Execution Modes

docs.google.com

111 pointsRooki51 comments

The dogma about preventing the sharing of pointers with C is a bit concerning. I've made use of that mechanism, to fantastic effect, albeit understanding the risks and consequences. The notion that it must be prevented because bad things can happen if you aren't careful isn't a starter, as there are essentially zero people doing that who've had problems. Instead a lot of people have enjoyed fantastic productivity because of the similarities of the implementations, and the synergy that allows.

But the team wants move to a compacting GC. Fine, add pin/unpin idioms. This ground has been well covered by other languages. Don't completely destroy a very productive mechanism because in some oddball cases it might not work.

I agree with the decision to forbid pointer sharing with C code.

This is the type of issues that nicely lead to security bugs, in the hands of the typical average enterprise developer.

Having a pin/unpin idiom might be good enough, though.

This is the type of issues that nicely lead to security bugs

It's also the type of functionality that enables the efficient use of high performance C libraries and existing code (I use it to great effect with the Intel compiler for extremely high performance SIMD code).

The "someone might not use it right, therefore it's anathema" notion is misguided, and is orthogonal with the philosophy of Go. Even C# and Java -- the pinnacle of "enterprise" languages -- understand and respect the notion of pointer sharing, and early on built in mechanisms to support it. Go is a league above, though, given that it follows C struct layout rules.

But I see where the person proposing banning it got their notion from -- it's a classic hubris of "if it isn't important to me, it isn't important to anyone". A compacting GC has benefits, but the lazy notion of wholesale blocking broad and powerful uses because one doesn't personally benefit from them is how languages die.

Java and .NET offer GC free allocation for such purposes, via their interop APIs. This is also one area that is being improved in Java 9 (Arrays 2.0, Value Types and the new FFI).

But as I mentioned, thus agreeing partially with you, pin/upin could be enough.

Although I would perfer pin/unpin,

You could easily implmement GC free allocation in Go and people do. Its not hard to write a library for explicit memory management:

1. mmap some memory 2. Write or port malloc and free like functions

This implementation could probably even use sync.Pool to good effect.

The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating. You may have used this successfully in the past, I know I have, and it works with the GC toolchain, but it is still technically incorrect to do so.

I think there's another design doc being worked on that will document the situation, and possibly have proposals to make sharing memory between Go and C more convenient. There's are many projects that make heavy use of this, where copying is slow or infeasible, and malloc is inconvenient.

The current state of the CGO interface is that you can't safely pass Go pointers to C code. That's all this document is repeating.

Where is this stated? Where is it said that it's technically incorrect?

Just to be clear, I'm not passing it and then running a thread in C that just runs with the pointer to an object that no longer exists in Go. Instead I'm calling a C function synchronously -- there is no possible way the element is going to be collected during that run. It could be compacted (though Go up until now has explicitly had a non-compacting GC, so not a current risk), of course, which is what brought up this issue, but that's the reason such interop languages have pinning (flagging that something shouldn't be moved).

It wasn't by chance -- it was by design, and it was purposefully understanding the lifetime of objects.

You're right, I can't find where it's explicitly forbidden in the docs, but it also isn't stated that you can pass pointers to C code either. There has been more recent discussion here (https://code.google.com/p/go/issues/detail?id=8310) and on the mailing lists too:

    > +rsc
    > Passing a Go pointer like that to C is problematic because eventually
    > we will want the garbage collector to be able to move things

and later on with a response from Dmitry:

    >> In my opinion, the current behavior should be covered by the 
    >> compatibility guarantee. In Go 1, passing pointers to Go-allocated 
    >> memory into C was not a problem. 
    >
    > This was never fully working. Only some individual cases were working.

Even though your C thread is being run synchronously, the GC is running concurrently in another thread (concurrent GC was introduced in go1.3), and has the ability to alter the memory that was passed to your C function.

The rule was never entirely clear, but there was definitely support towards allowing Go pointers to be visible in C code, as long as they were held referenced inside Go code somewhere. See this thread, for example:

https://code.google.com/p/go/issues/detail?id=6397#c11

That said, the rule is clearly changing now, and it is not even clear to what it is changing yet (see Ian's comment in this HN thread).

>Where is this stated? Where is it said that it's technically incorrect?

If I am not mistaken, it was discussed on the go-dev group.

Currently on the phone, cannot search properly for it.

EDIT: Now at home. This was the discussion thread

https://groups.google.com/forum/#!topic/golang-dev/pIuOcqAlv...

There is a bit more background at http://golang.org/issue/8310.

The problem is that we hope to move toward a concurrent GC. That means that GC must be able to run concurrently with a long-running cgo call--otherwise a call to some C function that never returns for perfectly valid reasons will block GC forever. And when we have a concurrent moving GC, it is no longer possible to casually pass a Go pointer to C.

In other words, we want to make the Go garbage collector work much better for Go programs, and we don't want calling C to prevent that.

Although the actual plan is not nailed down, I suspect that we will permit passing a Go pointer to C as long as the memory to which that pointer points does not itself contain any Go pointers. The cgo interface will be permitted to either pin the pointer for the duration of the call, or to copy the data to a pinned memory area--your program won't know which will happen.

If you need something more complex to work, you will have to allocate the memory on the C side. It's always OK to pass C pointers to C.

That's a much more reasonable proposal than the prior suggestions of a strict no-Go-pointers-into-C rule, and it'll keep most interfaces working. Some C packages such as qml will still need to change as they allow custom types to travel through C and back into Go, and these pointers may move, but that's certainly not the common case.

You can do your own pinning, which is what I do in gobind[1]. The basic idea is to keep a map in Go of the pointers:

    var ptrs = make(map[uintptr]unsafe.Pointer)

When passing a pointer from Go to C:

    var p unsafe.Pointer = ...
    id := uintptr(p)
    ptrs[id] = p
    C.Fn(id)

When C returns the pointer to Go to be used, run it through the ptrs map again

    //export GoFn
    func GoFn(id uintptr) {
        p := ptrs[id]
        // ... use p
    }

Don't forget a cleanup function where you delete(ptrs, id). If you want to pass the same pointer multiple times to C and keep it comparable, you'll need a second map.

All of this is requires being very careful, but the hard part is the notion of holding references to memory outside the realm of the GC. I don't believe a runtime-assisted pinning API can do any better than what you can do yourself with a map.

[1] https://godoc.org/code.google.com/p/go.mobile/cmd/gobind

I don't think this would really handle the pinning aspect if a compacting GC is implemented, would it? It does keep the pointer alive, but it technically doesn't do anything to prevent the GC from changing the pointer.

The uintptr version of the pointer is not moved by the GC, so it acts as a stable identifier.

This does not work if the GC updates the pointer concurrently.

The GC will not update a pointer once it is converted to the type uintptr. It becomes invisible to the precise collector, so the uintptr acts as an id value.

Worse, it might be collected then as the collector might think it is no longer in use.

But it's in the map, so it's in use.

This remark is what triggered my comment

> It becomes invisible to the precise collector

So does the conversion to uintptr remove it from the root lists the GC searches?

It's a map[uintptr]unsafe.Pointer. The key is invisible to the collector, but the value is an unsafe.Pointer, which is will be managed for you by the precise collector.

Ah, stupid me! Thanks for clarifying it.

That's a fine model when you want to hand a reference to a Go value, just to be used in Go itself, but it's not actual pinning. The data can still move, which means you cannot use the data in C, for example to share a buffer without copying into another temporary buffer managed by the C allocator.

That's right, you cannot dereference the pointer from C. If you want to use allocated memory from both C and Go, you'll need to allocate it in C.

What's the use-case for passing a pointer to C that can't be de-referenced?

You can pass it back into Go for dereferencing, at which point you can use the uintptr => unsafe.Pointer map, and it will be correct since the moving GC will have preserved the unsafe.Pointer address at the right location.

The pattern is useful. It just doesn't handle the most common case, which is passing a buffer or a simple output parameter into a C function.

Please note that the details aren't defined yet, and Ian indicates in a comment in this HN thread that some sort of pinning may exist. I also expect that to happen, given that a strict rule would break a lot of packages and turn what is today trivial into boring and slow code.

This would be awesome to have. +1!

With InfluxDB we want to give users the ability to define custom functions and it looks like this could be used to let people do that in Go or any language really. Whereas right now we're limited to doing something like using LuaJIT.

I would love to see the gc compiler support building dynamically linked executables against a libgo runtime Ala gccgo. Unless I'm misreading, this case doesn't seem covered. I love the standalone binary deployment ease as of now, but on a machine with tens of go applications, the binary size redundancy gets old. Or did I miss something?

We're in an age where everybody is wasting gigabytes of memory through virtualization, and nobody cares. Would those few megabytes of memory saved by dynamically linking against libgo really be worth it? For past 5 years or so, it seems people value ease of deployment much more than efficiency.

The people who value their memory usage just aren't using that kind of system, so there's some selection bias there.

You have a point, but on mobile and embedded devices, nobody is "wasting gigabytes of memory through virtualization" and not caring.

It doesn't cost gigabytes of RAM, but Android applications do run on a Java virtual machine.

But I'm being massively pedantic.

yes. +1

How large are the binaries you deploy? Unless it's hundreds of MB per binary I does see the dynamically linked executables being worth the trouble.

Of cause options are nice, I just worry that in the end I don't get to choose.

I think one of the motivations is that they want Go to be a language suitable for Android development and other use cases. In mobile and embedded devices, binary size still matters.

I believe that use case is covered via -buildmode=shared and -linkshared. Just run -buildmode=shared when building the standard library.

I think the more relevant news in there for most is the support for dynamic/runtime loading. The title here makes it sound like it's just for dynamic linking but I read the article to article to be sure and was pleasantly surprised :)

Good point, I updated the title to include the actual title of the doc but left the rest to avoid confusion.

On topic: I'm really excited to see this, even though it's made clear this is just a plan and not a promise it's still great news for things like Android development.

> Building Go packages as a shared library

Yes, please! We have a few potential Go projects which are blocked by the inability to separate Go code into plugins.

I have toyed with the idea of forking child processes and implementing plugins via pipe I/O, but that's a very unfriendly API interface, and I don't feel very inspired. (Go isn't that great at POSIX stuff like forking, either.)

Why do the plugins have to be dynamically loaded? Would a source-based plugin/package system be an alternative?

Consider a data processing app. It calls little data-processing modules written in Go. You want to open-source it because it's general-purpose and reusable; but you don't want to open-source all your in-house, proprietary data-processing components, too; they're not of use to anyone but yourself, or they may be something you can't open-source for other reasons.

That means the core project cannot list its plugins as dependencies. You will instead want to be able to load plugins from a configuration file.

(The problem is actually equally awkward in languages that have package systems, such as Node and Ruby. In Ruby, for example, a program can only see gems declared in the Gemfile; same problem with npm-shrinkwrap.json.)

I suppose you could set up a special build system where plugins are expected to be installed into a certain folder so that they're statically built in, with some kind of hook so that they're registered into the core. That's the Nginx approach. I'm not a big fan, especially since this method of building things need to be reinvented for each app that needs a plugin system.

You could, of course, invert the relationship and make this system a library: To set it up with your plugins, you write a small program that runs the system's main entrypoint with a list of plugins. But that just makes it harder to configure, build, use and deploy.

I realize it's just an early draft, but it's interesting (alarming?) that the proposed plugin API has no facilities for plugin discovery or separate namespaces. Nor any apparent ability for the plugin consumer to introspect into plugins.

I suppose the namespace issue can be mitigated by a strict naming convention (something like "[application_name].[plugin_name]"), but the others are kind of a bummer.

I don't have a whole lot of positive things to say about the Python setuptools API (perhaps better called "lack-of-API internal interface"), but at least it's possible to dynamically load python plugins from arbitrary locations in the filesystem and programmatically discover their contents.

I'm not sure I see the benefit of plugin discovery or separate namespaces. Loading a plugin gives you a Plugin struct, which is a namespace to lookup symbols (functions + globals) from that plugin.

Most plugins would I think by necessity be application specific, so I'm not sure how much gain there in in some sort of plugin discovery. But the process of loading a plugin would also allow the plugin to call into your code to register handlers & data structures in the main application through init() functions.

From a security perspective, executing arbitrary code in your address space in order to discover the contents of a plugin sounds ill-advised.

To me, it looks like they're building the low level building blocks that allow a number of different higher level APIs to be built, which is pretty neat.

The plugin API uses file names, so discovery would be at a higher level. Go packages impose their own namespace, so there shouldn't be any namespace issues. Introspection is useful, but would be a matter of examining the files using existing packages like debug/elf; it would not be part of the plugin API.

Basically the plugin API is isomorphic to dlopen/dlclose. Everything else is built on top of that.

OK, if true that makes sense. I was interpreting the Name parameter to Open() as being something like a global plugin name rather than a filesystem location.

That combined with the Plugin struct (which hopefully would have something like an iterable Map of exported symbols) resolves the issues I was concerned about. Nice.

Is there a mirror? For those of us in the corporate wasteland this is not accesible

Interesting that it will allow compilation of PIE executables. This is a mandatory requirement for any binaries run on newer versions of Android (L+).

Is it for security reasons? I know that OpenBSD uses PIEs so that the runtime can randomized address space locations.

The doc doesn't mention a timeline or Go version when we may see the new planned features.

That's unknown. Work will likely begin in December after Go 1.4 is released, so the earliest we'll see this work is in Go 1.5.

Reply on news.ycombinator.com